[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349162370577-12320-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:12:42 dist-test-slave-p468 krb5kdc[12327](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:12:42 dist-test-slave-p468 krb5kdc[12327](info): set up 2 sockets
Sep 25 15:12:42 dist-test-slave-p468 krb5kdc[12327](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260925 15:12:43.499653 12320 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.007s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349162370577-12320-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349162370577-12320-0/krb5kdc/test-user.keytab.
Sep 25 15:12:43 dist-test-slave-p468 krb5kdc[12327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:12:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:12:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.8.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab.
Entry for principal kudu/127.12.8.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab.
I20260925 15:12:43.668653 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349162370577-12320-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349162370577-12320-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349162370577-12320-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.62:41007
--webserver_interface=127.12.8.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab
--principal=kudu/127.12.8.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:42385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.62:41007
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:12:43.996312 12340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:12:43.996654 12340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:12:43.996816 12340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:12:44.004029 12340 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:12:44.004195 12340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:12:44.004307 12340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:12:44.004411 12340 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:12:44.004496 12340 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:12:44.014305 12340 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:42385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349162370577-12320-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349162370577-12320-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.8.62:41007
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab
--principal=kudu/127.12.8.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.62:41007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.8.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349162370577-12320-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:12:44.015662 12340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:12:44.017573 12340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:12:44.029515 12346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:12:44.029615 12345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:12:44.029726 12348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:12:44.030298 12340 server_base.cc:1034] running on GCE node
I20260925 15:12:44.030998 12340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:12:44.032786 12340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:12:44.034021 12340 hybrid_clock.cc:648] HybridClock initialized: now 1790349164033943 us; error 43 us; skew 500 ppm
I20260925 15:12:44.034406 12340 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:12:44 dist-test-slave-p468 krb5kdc[12327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349164, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:12:44.037532 12340 init.cc:377] Logged in from keytab as kudu/127.12.8.62@KRBTEST.COM (short username kudu)
I20260925 15:12:44.039045 12340 webserver.cc:460] Webserver started at http://127.12.8.62:42871/ using document root <none> and password file <none>
I20260925 15:12:44.039738 12340 fs_manager.cc:359] Metadata directory not provided
I20260925 15:12:44.039882 12340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:12:44.040176 12340 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:12:44.043478 12340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349162370577-12320-0/minicluster-data/master-0/data/instance:
uuid: "e1930b1878d04ddc832ac2495618b571"
format_stamp: "Formatted at 2026-09-25 15:12:44 on dist-test-slave-p468"
server_key: "c63207f85877e1cc2fac8b415fca4b1f"
server_key_iv: "5994d973d5ff34dcf6245787d1568fac"
server_key_version: "encryptionkey@0"
I20260925 15:12:44.044253 12340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349162370577-12320-0/minicluster-data/master-0/wal/instance:
uuid: "e1930b1878d04ddc832ac2495618b571"
format_stamp: "Formatted at 2026-09-25 15:12:44 on dist-test-slave-p468"
server_key: "c63207f85877e1cc2fac8b415fca4b1f"
server_key_iv: "5994d973d5ff34dcf6245787d1568fac"
server_key_version: "encryptionkey@0"
I20260925 15:12:44.049427 12340 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 15:12:44.052879 12355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:12:44.054126 12340 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 15:12:44.054327 12340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349162370577-12320-0/minicluster-data/master-0/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349162370577-12320-0/minicluster-data/master-0/wal
uuid: "e1930b1878d04ddc832ac2495618b571"
format_stamp: "Formatted at 2026-09-25 15:12:44 on dist-test-slave-p468"
server_key: "c63207f85877e1cc2fac8b415fca4b1f"
server_key_iv: "5994d973d5ff34dcf6245787d1568fac"
server_key_version: "encryptionkey@0"
I20260925 15:12:44.054510 12340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349162370577-12320-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349162370577-12320-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349162370577-12320-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:12:44.067849 12340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:12:44.071997 12340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:12:44.072263 12340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:12:44.090274 12340 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.62:41007
I20260925 15:12:44.090359 12406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.62:41007 every 8 connection(s)
I20260925 15:12:44.091573 12340 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
I20260925 15:12:44.094823 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 12340
I20260925 15:12:44.095148 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349162370577-12320-0/minicluster-data/master-0/wal/instance
I20260925 15:12:44.095750 12407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:12:44.097674 12320 external_mini_cluster.cc:1428] Setting key ec182dd2725dcbe60586a16b75e06135
I20260925 15:12:44.107981 12407 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1930b1878d04ddc832ac2495618b571: Bootstrap starting.
I20260925 15:12:44.110802 12407 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e1930b1878d04ddc832ac2495618b571: Neither blocks nor log segments found. Creating new log.
I20260925 15:12:44.111887 12407 log.cc:824] T 00000000000000000000000000000000 P e1930b1878d04ddc832ac2495618b571: Log is configured to *not* fsync() on all Append() calls
I20260925 15:12:44.114773 12407 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1930b1878d04ddc832ac2495618b571: No bootstrap required, opened a new log
I20260925 15:12:44.121827 12407 raft_consensus.cc:348] T 00000000000000000000000000000000 P e1930b1878d04ddc832ac2495618b571 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1930b1878d04ddc832ac2495618b571" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 41007 } }
I20260925 15:12:44.122188 12407 raft_consensus.cc:374] T 00000000000000000000000000000000 P e1930b1878d04ddc832ac2495618b571 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:12:44.122337 12407 raft_consensus.cc:729] T 00000000000000000000000000000000 P e1930b1878d04ddc832ac2495618b571 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1930b1878d04ddc832ac2495618b571, State: Initialized, Role: FOLLOWER
I20260925 15:12:44.122968 12407 consensus_queue.cc:260] T 00000000000000000000000000000000 P e1930b1878d04ddc832ac2495618b571 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1930b1878d04ddc832ac2495618b571" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 41007 } }
I20260925 15:12:44.123214 12407 raft_consensus.cc:388] T 00000000000000000000000000000000 P e1930b1878d04ddc832ac2495618b571 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:12:44.123360 12407 raft_consensus.cc:482] T 00000000000000000000000000000000 P e1930b1878d04ddc832ac2495618b571 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:12:44.123560 12407 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e1930b1878d04ddc832ac2495618b571 [term 0 FOLLOWER]: Advancing to term 1
Sep 25 15:12:44 dist-test-slave-p468 krb5kdc[12327](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
I20260925 15:12:44.126011 12407 raft_consensus.cc:504] T 00000000000000000000000000000000 P e1930b1878d04ddc832ac2495618b571 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1930b1878d04ddc832ac2495618b571" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 41007 } }
I20260925 15:12:44.126552 12407 leader_election.cc:302] T 00000000000000000000000000000000 P e1930b1878d04ddc832ac2495618b571 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e1930b1878d04ddc832ac2495618b571; no voters:
I20260925 15:12:44.127158 12407 leader_election.cc:288] T 00000000000000000000000000000000 P e1930b1878d04ddc832ac2495618b571 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:12:44.127297 12412 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e1930b1878d04ddc832ac2495618b571 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:12:44.129350 12407 sys_catalog.cc:564] T 00000000000000000000000000000000 P e1930b1878d04ddc832ac2495618b571 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:12:44.130447 12412 raft_consensus.cc:686] T 00000000000000000000000000000000 P e1930b1878d04ddc832ac2495618b571 [term 1 LEADER]: Becoming Leader. State: Replica: e1930b1878d04ddc832ac2495618b571, State: Running, Role: LEADER
I20260925 15:12:44.131067 12412 consensus_queue.cc:237] T 00000000000000000000000000000000 P e1930b1878d04ddc832ac2495618b571 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1930b1878d04ddc832ac2495618b571" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 41007 } }
I20260925 15:12:44.134057 12413 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1930b1878d04ddc832ac2495618b571 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e1930b1878d04ddc832ac2495618b571" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1930b1878d04ddc832ac2495618b571" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 41007 } } }
I20260925 15:12:44.134492 12413 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1930b1878d04ddc832ac2495618b571 [sys.catalog]: This master's current role is: LEADER
I20260925 15:12:44.134495 12414 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1930b1878d04ddc832ac2495618b571 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1930b1878d04ddc832ac2495618b571. Latest consensus state: current_term: 1 leader_uuid: "e1930b1878d04ddc832ac2495618b571" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1930b1878d04ddc832ac2495618b571" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 41007 } } }
I20260925 15:12:44.134817 12414 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1930b1878d04ddc832ac2495618b571 [sys.catalog]: This master's current role is: LEADER
I20260925 15:12:44.139971 12410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:44.102289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54024 (local address 127.12.8.62:41007)
0925 15:12:44.104361 (+ 2072us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:44.104375 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:44.104425 (+ 50us) server_negotiation.cc:407] Connection header received
0925 15:12:44.112963 (+ 8538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:44.112993 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:44.113411 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:44.113763 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:44.116630 (+ 2867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:44.117977 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:44.119622 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:44.120289 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:44.120501 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:44.126432 (+ 5931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:44.126440 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:44.126454 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:12:44.126486 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:44.131396 (+ 4910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:44.132393 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:44.132405 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:44.132416 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:44.132662 (+ 246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:44.133239 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:44.133246 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:44.133253 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:44.134070 (+ 817us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:44.137561 (+ 3491us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:44.137864 (+ 303us) server_negotiation.cc:299] Negotiation successful
0925 15:12:44.138382 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1824,"thread_start_us":1598,"threads_started":1}
I20260925 15:12:44.146893 12419 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:12:44.151997 12419 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:12:44.160444 12419 catalog_manager.cc:1347] Generated new cluster ID: 226b2583da974389ad0be4bc94a44655
I20260925 15:12:44.160578 12419 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:12:44.209489 12419 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:12:44.211357 12419 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:12:44.218788 12419 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e1930b1878d04ddc832ac2495618b571: Generated new TSK 0
I20260925 15:12:44.219794 12419 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.8.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.1.keytab.
Entry for principal kudu/127.12.8.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.1.keytab.
I20260925 15:12:44.269204 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349162370577-12320-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349162370577-12320-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349162370577-12320-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349162370577-12320-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.1:0
--local_ip_for_outbound_sockets=127.12.8.1
--webserver_interface=127.12.8.1
--webserver_port=0
--tserver_master_addrs=127.12.8.62:41007
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.1.keytab
--principal=kudu/127.12.8.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:42385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:12:44.579684 12433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:12:44.579946 12433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:12:44.580040 12433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:12:44.587033 12433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:12:44.587136 12433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:12:44.587246 12433 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.1
I20260925 15:12:44.596450 12433 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:42385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349162370577-12320-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349162370577-12320-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.1.keytab
--principal=kudu/127.12.8.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349162370577-12320-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.8.1
--webserver_port=0
--tserver_master_addrs=127.12.8.62:41007
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.1
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349162370577-12320-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:12:44.597615 12433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:12:44.598853 12433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:12:44.611790 12441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:12:44.611876 12438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:12:44.611851 12439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:12:44.612712 12433 server_base.cc:1034] running on GCE node
I20260925 15:12:44.613289 12433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:12:44.614176 12433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:12:44.615432 12433 hybrid_clock.cc:648] HybridClock initialized: now 1790349164615377 us; error 51 us; skew 500 ppm
I20260925 15:12:44.615742 12433 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:12:44 dist-test-slave-p468 krb5kdc[12327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349164, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:12:44.618851 12433 init.cc:377] Logged in from keytab as kudu/127.12.8.1@KRBTEST.COM (short username kudu)
I20260925 15:12:44.620260 12433 webserver.cc:460] Webserver started at http://127.12.8.1:36909/ using document root <none> and password file <none>
I20260925 15:12:44.620913 12433 fs_manager.cc:359] Metadata directory not provided
I20260925 15:12:44.621064 12433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:12:44.621373 12433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:12:44.624328 12433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349162370577-12320-0/minicluster-data/ts-0/data/instance:
uuid: "efeaec988e3e4bc4b16212e1bf18135e"
format_stamp: "Formatted at 2026-09-25 15:12:44 on dist-test-slave-p468"
server_key: "c6291cbfc9fd222ddd16dd433ebe5b39"
server_key_iv: "60b1450b9b577ca52590a824d6e29b3c"
server_key_version: "encryptionkey@0"
I20260925 15:12:44.625111 12433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349162370577-12320-0/minicluster-data/ts-0/wal/instance:
uuid: "efeaec988e3e4bc4b16212e1bf18135e"
format_stamp: "Formatted at 2026-09-25 15:12:44 on dist-test-slave-p468"
server_key: "c6291cbfc9fd222ddd16dd433ebe5b39"
server_key_iv: "60b1450b9b577ca52590a824d6e29b3c"
server_key_version: "encryptionkey@0"
I20260925 15:12:44.630339 12433 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260925 15:12:44.633895 12448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:12:44.635365 12433 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:12:44.635557 12433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349162370577-12320-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349162370577-12320-0/minicluster-data/ts-0/wal
uuid: "efeaec988e3e4bc4b16212e1bf18135e"
format_stamp: "Formatted at 2026-09-25 15:12:44 on dist-test-slave-p468"
server_key: "c6291cbfc9fd222ddd16dd433ebe5b39"
server_key_iv: "60b1450b9b577ca52590a824d6e29b3c"
server_key_version: "encryptionkey@0"
I20260925 15:12:44.635766 12433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349162370577-12320-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349162370577-12320-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349162370577-12320-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:12:44.648612 12433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:12:44.652593 12433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:12:44.652890 12433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:12:44.653789 12433 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:12:44.655114 12433 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:12:44.655216 12433 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:12:44.655361 12433 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:12:44.655439 12433 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:12:44.688449 12433 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.1:41143
I20260925 15:12:44.688498 12560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.1:41143 every 8 connection(s)
I20260925 15:12:44.689697 12433 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349162370577-12320-0/minicluster-data/ts-0/data/info.pb
I20260925 15:12:44.694353 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 12433
I20260925 15:12:44.694585 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349162370577-12320-0/minicluster-data/ts-0/wal/instance
I20260925 15:12:44.695180 12320 external_mini_cluster.cc:1428] Setting key ec033695e3d70807f73cf76914947113
Sep 25 15:12:44 dist-test-slave-p468 krb5kdc[12327](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349164, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.1@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
I20260925 15:12:44.709070 12563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:44.692197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.1:60655 (local address 127.12.8.62:41007)
0925 15:12:44.692588 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:44.692592 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:44.693564 (+ 972us) server_negotiation.cc:407] Connection header received
0925 15:12:44.695467 (+ 1903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:44.695473 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:44.695560 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:44.695682 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:44.697372 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:44.698049 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:44.699580 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:44.699983 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:44.700116 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:44.703521 (+ 3405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:44.703526 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:44.703529 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:12:44.703558 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:44.706048 (+ 2490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:44.706815 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:44.706820 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:44.706822 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:44.706939 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:44.707366 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:44.707373 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:44.707381 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:44.707644 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:44.707799 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:44.708464 (+ 665us) server_negotiation.cc:299] Negotiation successful
0925 15:12:44.708693 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":192,"threads_started":1}
WARNING: no policy specified for kudu/127.12.8.2@KRBTEST.COM; defaulting to no policy
I20260925 15:12:44.710057 12564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:44.692741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:41007 (local address 127.12.8.1:60655)
0925 15:12:44.693402 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:44.693437 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 15:12:44.695172 (+ 1735us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:12:44.695923 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:12:44.695931 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:12:44.696580 (+ 649us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:12:44.697212 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:44.697230 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:44.698215 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:44.698222 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:44.699435 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:44.699446 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:44.700217 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:44.700225 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:44.700442 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:44.701130 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:12:44.701153 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:12:44.703296 (+ 2143us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:12:44.706218 (+ 2922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:12:44.706228 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:12:44.706243 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:12:44.706670 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:12:44.707053 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:12:44.707058 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:12:44.707062 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:12:44.707246 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:12:44.707773 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:12:44.707780 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:12:44.708114 (+ 334us) client_negotiation.cc:769] Sending connection context
0925 15:12:44.708376 (+ 262us) client_negotiation.cc:240] Negotiation successful
0925 15:12:44.708904 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":253,"threads_started":1}
I20260925 15:12:44.711622 12561 heartbeater.cc:347] Connected to a master server at 127.12.8.62:41007
I20260925 15:12:44.711922 12561 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.2@KRBTEST.COM" created.
I20260925 15:12:44.712648 12561 heartbeater.cc:511] Master 127.12.8.62:41007 requested a full tablet report, sending...
I20260925 15:12:44.714674 12372 ts_manager.cc:194] Registered new tserver with Master: efeaec988e3e4bc4b16212e1bf18135e (127.12.8.1:41143)
I20260925 15:12:44.716574 12372 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.1@KRBTEST.COM'} at 127.12.8.1:60655
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.2.keytab.
Entry for principal kudu/127.12.8.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.2.keytab.
I20260925 15:12:44.733309 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349162370577-12320-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349162370577-12320-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349162370577-12320-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349162370577-12320-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.2:0
--local_ip_for_outbound_sockets=127.12.8.2
--webserver_interface=127.12.8.2
--webserver_port=0
--tserver_master_addrs=127.12.8.62:41007
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.2.keytab
--principal=kudu/127.12.8.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:42385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:12:45.064627 12567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:12:45.064884 12567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:12:45.065050 12567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:12:45.072934 12567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:12:45.073064 12567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:12:45.073391 12567 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.2
I20260925 15:12:45.083195 12567 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:42385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349162370577-12320-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349162370577-12320-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.2.keytab
--principal=kudu/127.12.8.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349162370577-12320-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.8.2
--webserver_port=0
--tserver_master_addrs=127.12.8.62:41007
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.2
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349162370577-12320-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:12:45.084444 12567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:12:45.085738 12567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:12:45.099041 12572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:12:45.098979 12573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:12:45.099623 12575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:12:45.107587 12567 server_base.cc:1034] running on GCE node
I20260925 15:12:45.108268 12567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:12:45.109200 12567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:12:45.110436 12567 hybrid_clock.cc:648] HybridClock initialized: now 1790349165110407 us; error 51 us; skew 500 ppm
I20260925 15:12:45.110738 12567 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:12:45 dist-test-slave-p468 krb5kdc[12327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349165, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:12:45.113646 12567 init.cc:377] Logged in from keytab as kudu/127.12.8.2@KRBTEST.COM (short username kudu)
I20260925 15:12:45.115075 12567 webserver.cc:460] Webserver started at http://127.12.8.2:43923/ using document root <none> and password file <none>
I20260925 15:12:45.115753 12567 fs_manager.cc:359] Metadata directory not provided
I20260925 15:12:45.115914 12567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:12:45.116225 12567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:12:45.119061 12567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349162370577-12320-0/minicluster-data/ts-1/data/instance:
uuid: "0d94595a5ed04631b530acd61ed80bc5"
format_stamp: "Formatted at 2026-09-25 15:12:45 on dist-test-slave-p468"
server_key: "f65327db8cbe2a667536eb342125fe9c"
server_key_iv: "997115f5502885c8775ab0a2c9ed3cb2"
server_key_version: "encryptionkey@0"
I20260925 15:12:45.119787 12567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349162370577-12320-0/minicluster-data/ts-1/wal/instance:
uuid: "0d94595a5ed04631b530acd61ed80bc5"
format_stamp: "Formatted at 2026-09-25 15:12:45 on dist-test-slave-p468"
server_key: "f65327db8cbe2a667536eb342125fe9c"
server_key_iv: "997115f5502885c8775ab0a2c9ed3cb2"
server_key_version: "encryptionkey@0"
I20260925 15:12:45.124531 12567 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 15:12:45.127779 12582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:12:45.129156 12567 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 15:12:45.129343 12567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349162370577-12320-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349162370577-12320-0/minicluster-data/ts-1/wal
uuid: "0d94595a5ed04631b530acd61ed80bc5"
format_stamp: "Formatted at 2026-09-25 15:12:45 on dist-test-slave-p468"
server_key: "f65327db8cbe2a667536eb342125fe9c"
server_key_iv: "997115f5502885c8775ab0a2c9ed3cb2"
server_key_version: "encryptionkey@0"
I20260925 15:12:45.129558 12567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349162370577-12320-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349162370577-12320-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349162370577-12320-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:12:45.155992 12567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:12:45.160526 12567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:12:45.160816 12567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:12:45.161809 12567 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:12:45.163374 12567 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:12:45.163499 12567 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:12:45.163666 12567 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:12:45.163745 12567 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:12:45.200407 12567 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.2:34781
I20260925 15:12:45.200441 12694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.2:34781 every 8 connection(s)
I20260925 15:12:45.201678 12567 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349162370577-12320-0/minicluster-data/ts-1/data/info.pb
I20260925 15:12:45.210142 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 12567
I20260925 15:12:45.210361 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349162370577-12320-0/minicluster-data/ts-1/wal/instance
I20260925 15:12:45.210901 12320 external_mini_cluster.cc:1428] Setting key dc790df1a694004c5f1cc11e0b0fd4b6
Sep 25 15:12:45 dist-test-slave-p468 krb5kdc[12327](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349165, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.2@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
I20260925 15:12:45.222774 12563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:45.204222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.2:38339 (local address 127.12.8.62:41007)
0925 15:12:45.204415 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:45.204422 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:45.205545 (+ 1123us) server_negotiation.cc:407] Connection header received
0925 15:12:45.207395 (+ 1850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:45.207400 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:45.207464 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:45.207555 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:45.208943 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:45.209652 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:45.212050 (+ 2398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:45.212451 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:45.212552 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:45.216788 (+ 4236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:45.216793 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:45.216797 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:12:45.216827 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:45.219652 (+ 2825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:45.220578 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:45.220582 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:45.220584 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:45.220713 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:45.221187 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:45.221191 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:45.221193 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:45.221430 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:45.221537 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:45.222388 (+ 851us) server_negotiation.cc:299] Negotiation successful
0925 15:12:45.222553 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 15:12:45.224386 12697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:45.204682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:41007 (local address 127.12.8.2:38339)
0925 15:12:45.205388 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:45.205426 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 15:12:45.207153 (+ 1727us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:12:45.207755 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:12:45.207763 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:12:45.208271 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:12:45.208801 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:45.208816 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:45.209822 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:45.209831 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:45.211747 (+ 1916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:45.211759 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:45.212693 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:45.212703 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:45.212925 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:45.213699 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:12:45.213734 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:12:45.216561 (+ 2827us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:12:45.219788 (+ 3227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:12:45.219800 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:12:45.219820 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:12:45.220444 (+ 624us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:12:45.220827 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:12:45.220835 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:12:45.220842 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:12:45.221073 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:12:45.221572 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:12:45.221582 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:12:45.222001 (+ 419us) client_negotiation.cc:769] Sending connection context
0925 15:12:45.222313 (+ 312us) client_negotiation.cc:240] Negotiation successful
0925 15:12:45.222901 (+ 588us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":295,"threads_started":1}
I20260925 15:12:45.226078 12695 heartbeater.cc:347] Connected to a master server at 127.12.8.62:41007
I20260925 15:12:45.226414 12695 heartbeater.cc:464] Registering TS with master...
I20260925 15:12:45.227133 12695 heartbeater.cc:511] Master 127.12.8.62:41007 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.12.8.3@KRBTEST.COM; defaulting to no policy
I20260925 15:12:45.228574 12372 ts_manager.cc:194] Registered new tserver with Master: 0d94595a5ed04631b530acd61ed80bc5 (127.12.8.2:34781)
I20260925 15:12:45.229602 12372 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.2@KRBTEST.COM'} at 127.12.8.2:38339
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.3.keytab.
Entry for principal kudu/127.12.8.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.3.keytab.
I20260925 15:12:45.251421 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349162370577-12320-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349162370577-12320-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349162370577-12320-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349162370577-12320-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.3:0
--local_ip_for_outbound_sockets=127.12.8.3
--webserver_interface=127.12.8.3
--webserver_port=0
--tserver_master_addrs=127.12.8.62:41007
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.3.keytab
--principal=kudu/127.12.8.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:42385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:12:45.566253 12700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:12:45.566522 12700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:12:45.566710 12700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:12:45.573887 12700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:12:45.574007 12700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:12:45.574246 12700 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.3
I20260925 15:12:45.584194 12700 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:42385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349162370577-12320-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349162370577-12320-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.3.keytab
--principal=kudu/127.12.8.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349162370577-12320-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.8.3
--webserver_port=0
--tserver_master_addrs=127.12.8.62:41007
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.3
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349162370577-12320-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:12:45.585495 12700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:12:45.586913 12700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:12:45.600114 12705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:12:45.600171 12706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:12:45.600427 12708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:12:45.601086 12700 server_base.cc:1034] running on GCE node
I20260925 15:12:45.601668 12700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:12:45.602659 12700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:12:45.603863 12700 hybrid_clock.cc:648] HybridClock initialized: now 1790349165603816 us; error 33 us; skew 500 ppm
I20260925 15:12:45.604221 12700 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:12:45 dist-test-slave-p468 krb5kdc[12327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349165, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:12:45.607455 12700 init.cc:377] Logged in from keytab as kudu/127.12.8.3@KRBTEST.COM (short username kudu)
I20260925 15:12:45.608938 12700 webserver.cc:460] Webserver started at http://127.12.8.3:37789/ using document root <none> and password file <none>
I20260925 15:12:45.609607 12700 fs_manager.cc:359] Metadata directory not provided
I20260925 15:12:45.609740 12700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:12:45.610059 12700 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:12:45.612944 12700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349162370577-12320-0/minicluster-data/ts-2/data/instance:
uuid: "36ef9a54bc9141eb9e8c290b0acfabf4"
format_stamp: "Formatted at 2026-09-25 15:12:45 on dist-test-slave-p468"
server_key: "c2c6826a5ba2590c27ab4e301a64f135"
server_key_iv: "c7cfc43fe325e388e482f88818962706"
server_key_version: "encryptionkey@0"
I20260925 15:12:45.613698 12700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349162370577-12320-0/minicluster-data/ts-2/wal/instance:
uuid: "36ef9a54bc9141eb9e8c290b0acfabf4"
format_stamp: "Formatted at 2026-09-25 15:12:45 on dist-test-slave-p468"
server_key: "c2c6826a5ba2590c27ab4e301a64f135"
server_key_iv: "c7cfc43fe325e388e482f88818962706"
server_key_version: "encryptionkey@0"
I20260925 15:12:45.618835 12700 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260925 15:12:45.622150 12715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:12:45.623481 12700 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 15:12:45.623660 12700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349162370577-12320-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349162370577-12320-0/minicluster-data/ts-2/wal
uuid: "36ef9a54bc9141eb9e8c290b0acfabf4"
format_stamp: "Formatted at 2026-09-25 15:12:45 on dist-test-slave-p468"
server_key: "c2c6826a5ba2590c27ab4e301a64f135"
server_key_iv: "c7cfc43fe325e388e482f88818962706"
server_key_version: "encryptionkey@0"
I20260925 15:12:45.623865 12700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349162370577-12320-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349162370577-12320-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349162370577-12320-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:12:45.653025 12700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:12:45.657096 12700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:12:45.657464 12700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:12:45.658545 12700 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:12:45.659832 12700 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:12:45.659936 12700 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:12:45.660094 12700 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:12:45.660167 12700 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:12:45.693703 12700 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.3:39579
I20260925 15:12:45.693738 12827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.3:39579 every 8 connection(s)
I20260925 15:12:45.695009 12700 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349162370577-12320-0/minicluster-data/ts-2/data/info.pb
I20260925 15:12:45.697336 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 12700
I20260925 15:12:45.697525 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349162370577-12320-0/minicluster-data/ts-2/wal/instance
I20260925 15:12:45.698021 12320 external_mini_cluster.cc:1428] Setting key e8eca840718873260d81641a304edb1f
Sep 25 15:12:45 dist-test-slave-p468 krb5kdc[12327](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349165, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.3@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
I20260925 15:12:45.715492 12563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:45.697612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.3:48715 (local address 127.12.8.62:41007)
0925 15:12:45.697824 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:45.697828 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:45.699208 (+ 1380us) server_negotiation.cc:407] Connection header received
0925 15:12:45.701447 (+ 2239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:45.701454 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:45.701543 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:45.701662 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:45.703279 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:45.704064 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:45.705637 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:45.706051 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:45.706182 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:45.709624 (+ 3442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:45.709629 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:45.709633 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:12:45.709665 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:45.712139 (+ 2474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:45.713039 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:45.713045 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:45.713048 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:45.713259 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:45.713783 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:45.713787 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:45.713789 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:45.714177 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:45.714358 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:45.715104 (+ 746us) server_negotiation.cc:299] Negotiation successful
0925 15:12:45.715263 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 15:12:45.716517 12830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:45.698120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:41007 (local address 127.12.8.3:48715)
0925 15:12:45.698988 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:45.699048 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0925 15:12:45.701138 (+ 2090us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:12:45.701887 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:12:45.701896 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:12:45.702491 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:12:45.703049 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:45.703065 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:45.704220 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:45.704224 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:45.705466 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:45.705478 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:45.706333 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:45.706338 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:45.706550 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:45.707200 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:12:45.707223 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:12:45.709423 (+ 2200us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:12:45.712272 (+ 2849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:12:45.712282 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:12:45.712299 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:12:45.712910 (+ 611us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:12:45.713386 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:12:45.713390 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:12:45.713393 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:12:45.713658 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:12:45.714340 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:12:45.714348 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:12:45.714773 (+ 425us) client_negotiation.cc:769] Sending connection context
0925 15:12:45.714994 (+ 221us) client_negotiation.cc:240] Negotiation successful
0925 15:12:45.715371 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":346,"threads_started":1}
I20260925 15:12:45.717912 12828 heartbeater.cc:347] Connected to a master server at 127.12.8.62:41007
I20260925 15:12:45.718317 12828 heartbeater.cc:464] Registering TS with master...
I20260925 15:12:45.719039 12828 heartbeater.cc:511] Master 127.12.8.62:41007 requested a full tablet report, sending...
I20260925 15:12:45.719936 12561 heartbeater.cc:503] Master 127.12.8.62:41007 was elected leader, sending a full tablet report...
I20260925 15:12:45.720571 12372 ts_manager.cc:194] Registered new tserver with Master: 36ef9a54bc9141eb9e8c290b0acfabf4 (127.12.8.3:39579)
I20260925 15:12:45.721582 12372 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.3@KRBTEST.COM'} at 127.12.8.3:48715
I20260925 15:12:45.726897 12320 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 15:12:45 dist-test-slave-p468 krb5kdc[12327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349165, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 15:12:45 dist-test-slave-p468 krb5kdc[12327](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349165, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.8.1@KRBTEST.COM
I20260925 15:12:45.771327 12834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:45.756138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36090 (local address 127.12.8.1:41143)
0925 15:12:45.756798 (+ 660us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:45.756805 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:45.756834 (+ 29us) server_negotiation.cc:407] Connection header received
0925 15:12:45.756964 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:45.756971 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:45.757247 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:45.757394 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:45.758941 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:45.760100 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:45.761332 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:45.761979 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:45.762092 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:45.765824 (+ 3732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:45.765835 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:45.765851 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:12:45.765889 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:45.768803 (+ 2914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:45.769386 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:45.769392 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:45.769395 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:45.769529 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:45.769868 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:45.769871 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:45.769873 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:45.770312 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:45.770539 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:45.770775 (+ 236us) server_negotiation.cc:299] Negotiation successful
0925 15:12:45.770972 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":347,"threads_started":1}
W20260925 15:12:45.772568 12476 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36090
Sep 25 15:12:45 dist-test-slave-p468 krb5kdc[12327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349165, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 15:12:45 dist-test-slave-p468 krb5kdc[12327](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349165, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.1@KRBTEST.COM
I20260925 15:12:45.815220 12834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:45.801753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36094 (local address 127.12.8.1:41143)
0925 15:12:45.801975 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:45.801982 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:45.802214 (+ 232us) server_negotiation.cc:407] Connection header received
0925 15:12:45.802440 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:45.802447 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:45.802520 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:45.802627 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:45.803932 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:45.804616 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:45.805809 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:45.806326 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:45.806453 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:45.810379 (+ 3926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:45.810384 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:45.810387 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:12:45.810418 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:45.812790 (+ 2372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:45.813400 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:45.813406 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:45.813409 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:45.813536 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:45.813885 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:45.813892 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:45.813895 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:45.814143 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:45.814392 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:45.814681 (+ 289us) server_negotiation.cc:299] Negotiation successful
0925 15:12:45.814891 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 15:12:45.816743 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 12433
I20260925 15:12:45.829406 12834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:45.817790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36098 (local address 127.12.8.1:41143)
0925 15:12:45.818000 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:45.818007 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:45.818108 (+ 101us) server_negotiation.cc:407] Connection header received
0925 15:12:45.818292 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:45.818296 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:45.818352 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:45.818435 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:45.819477 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:45.820203 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:45.821356 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:45.821805 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:45.821900 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:45.824044 (+ 2144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:45.824049 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:45.824052 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:12:45.824081 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:45.826717 (+ 2636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:45.827467 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:45.827471 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:45.827473 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:45.827613 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:45.827951 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:45.827954 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:45.827957 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:45.828254 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:45.828503 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:45.828775 (+ 272us) server_negotiation.cc:299] Negotiation successful
0925 15:12:45.828925 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 15:12:45.830220 12556 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:12:45.915223 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 12433
I20260925 15:12:45.930984 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 12567
Sep 25 15:12:45 dist-test-slave-p468 krb5kdc[12327](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349165, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.2@KRBTEST.COM
I20260925 15:12:45.946940 12842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:45.931938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34810 (local address 127.12.8.2:34781)
0925 15:12:45.932635 (+ 697us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:45.932642 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:45.932666 (+ 24us) server_negotiation.cc:407] Connection header received
0925 15:12:45.932724 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:45.932730 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:45.932929 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:45.933052 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:45.934385 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:45.935403 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:45.936644 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:45.937214 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:45.937308 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:45.941377 (+ 4069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:45.941386 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:45.941399 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:12:45.941435 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:45.944204 (+ 2769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:45.944853 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:45.944861 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:45.944863 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:45.944972 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:45.945360 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:45.945367 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:45.945370 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:45.945697 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:45.946036 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:45.946316 (+ 280us) server_negotiation.cc:299] Negotiation successful
0925 15:12:45.946587 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":402,"threads_started":1}
I20260925 15:12:45.948203 12690 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:12:46.033821 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 12567
I20260925 15:12:46.049038 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 12700
Sep 25 15:12:46 dist-test-slave-p468 krb5kdc[12327](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349165, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.3@KRBTEST.COM
I20260925 15:12:46.065752 12844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:46.050024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48622 (local address 127.12.8.3:39579)
0925 15:12:46.050741 (+ 717us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:46.050747 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:46.050776 (+ 29us) server_negotiation.cc:407] Connection header received
0925 15:12:46.050835 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:46.050841 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:46.051098 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:46.051290 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:46.052567 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:46.053599 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:46.055167 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:46.055731 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:46.055867 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:46.060184 (+ 4317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:46.060195 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:46.060210 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:12:46.060244 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:46.063040 (+ 2796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:46.063721 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:46.063727 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:46.063730 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:46.063854 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:46.064220 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:46.064226 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:46.064229 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:46.064550 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:46.064869 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:46.065170 (+ 301us) server_negotiation.cc:299] Negotiation successful
0925 15:12:46.065425 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":422,"threads_started":1}
I20260925 15:12:46.067090 12823 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:12:46.155081 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 12700
I20260925 15:12:46.170429 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 12340
I20260925 15:12:46.171586 12402 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:12:46.272009 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 12340
2026-09-25T15:12:46Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3868 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349162370577-12320-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:12:46 dist-test-slave-p468 krb5kdc[12849](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:12:46 dist-test-slave-p468 krb5kdc[12849](info): set up 2 sockets
Sep 25 15:12:46 dist-test-slave-p468 krb5kdc[12849](info): commencing operation
krb5kdc: starting...
W20260925 15:12:47.349987 12320 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.002s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349162370577-12320-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349162370577-12320-0/krb5kdc/test-user.keytab.
Sep 25 15:12:47 dist-test-slave-p468 krb5kdc[12849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:12:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:12:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.8.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab.
Entry for principal kudu/127.12.8.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab.
I20260925 15:12:47.521462 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349162370577-12320-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349162370577-12320-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349162370577-12320-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.62:40783
--webserver_interface=127.12.8.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab
--principal=kudu/127.12.8.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:40187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.62:40783
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:12:47.836223 12862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:12:47.836602 12862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:12:47.836707 12862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:12:47.843597 12862 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:12:47.843698 12862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:12:47.843742 12862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:12:47.843792 12862 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:12:47.843837 12862 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:12:47.853199 12862 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:40187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349162370577-12320-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349162370577-12320-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.8.62:40783
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab
--principal=kudu/127.12.8.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.62:40783
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.8.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349162370577-12320-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:12:47.854508 12862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:12:47.855715 12862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:12:47.869590 12867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:12:47.869680 12870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:12:47.869658 12868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:12:47.870409 12862 server_base.cc:1034] running on GCE node
I20260925 15:12:47.871176 12862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:12:47.872467 12862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:12:47.873700 12862 hybrid_clock.cc:648] HybridClock initialized: now 1790349167873656 us; error 57 us; skew 500 ppm
I20260925 15:12:47.874044 12862 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:12:47 dist-test-slave-p468 krb5kdc[12849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349167, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:12:47.881639 12862 init.cc:377] Logged in from keytab as kudu/127.12.8.62@KRBTEST.COM (short username kudu)
I20260925 15:12:47.883234 12862 webserver.cc:460] Webserver started at http://127.12.8.62:33351/ using document root <none> and password file <none>
I20260925 15:12:47.883922 12862 fs_manager.cc:359] Metadata directory not provided
I20260925 15:12:47.884061 12862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:12:47.884359 12862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:12:47.887315 12862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349162370577-12320-0/minicluster-data/master-0/data/instance:
uuid: "6f5813fa68024a9b9509f46024db9a41"
format_stamp: "Formatted at 2026-09-25 15:12:47 on dist-test-slave-p468"
server_key: "a59a1b6d490650a7d3f8bcce2f5426a5"
server_key_iv: "04824d75650447246742daaca8622dc7"
server_key_version: "encryptionkey@0"
I20260925 15:12:47.888058 12862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349162370577-12320-0/minicluster-data/master-0/wal/instance:
uuid: "6f5813fa68024a9b9509f46024db9a41"
format_stamp: "Formatted at 2026-09-25 15:12:47 on dist-test-slave-p468"
server_key: "a59a1b6d490650a7d3f8bcce2f5426a5"
server_key_iv: "04824d75650447246742daaca8622dc7"
server_key_version: "encryptionkey@0"
I20260925 15:12:47.893163 12862 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260925 15:12:47.896688 12877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:12:47.898073 12862 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 15:12:47.898322 12862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349162370577-12320-0/minicluster-data/master-0/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349162370577-12320-0/minicluster-data/master-0/wal
uuid: "6f5813fa68024a9b9509f46024db9a41"
format_stamp: "Formatted at 2026-09-25 15:12:47 on dist-test-slave-p468"
server_key: "a59a1b6d490650a7d3f8bcce2f5426a5"
server_key_iv: "04824d75650447246742daaca8622dc7"
server_key_version: "encryptionkey@0"
I20260925 15:12:47.898548 12862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349162370577-12320-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349162370577-12320-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349162370577-12320-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:12:47.914428 12862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:12:47.919097 12862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:12:47.919414 12862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:12:47.939308 12862 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.62:40783
I20260925 15:12:47.939355 12928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.62:40783 every 8 connection(s)
I20260925 15:12:47.940779 12862 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
I20260925 15:12:47.944479 12929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:12:47.947557 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 12862
I20260925 15:12:47.947798 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349162370577-12320-0/minicluster-data/master-0/wal/instance
I20260925 15:12:47.948603 12320 external_mini_cluster.cc:1428] Setting key 8fb03147632c7a8df9d296e4057e0c8f
I20260925 15:12:47.957496 12929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f5813fa68024a9b9509f46024db9a41: Bootstrap starting.
I20260925 15:12:47.961385 12929 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f5813fa68024a9b9509f46024db9a41: Neither blocks nor log segments found. Creating new log.
I20260925 15:12:47.962800 12929 log.cc:824] T 00000000000000000000000000000000 P 6f5813fa68024a9b9509f46024db9a41: Log is configured to *not* fsync() on all Append() calls
Sep 25 15:12:47 dist-test-slave-p468 krb5kdc[12849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
I20260925 15:12:47.965859 12929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f5813fa68024a9b9509f46024db9a41: No bootstrap required, opened a new log
I20260925 15:12:47.972230 12932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:47.950754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42180 (local address 127.12.8.62:40783)
0925 15:12:47.951547 (+ 793us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:47.951559 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:47.951602 (+ 43us) server_negotiation.cc:407] Connection header received
0925 15:12:47.953283 (+ 1681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:47.953306 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:47.953722 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:47.954060 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:47.956000 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:47.957278 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:47.959023 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:47.959707 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:47.959838 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:47.964813 (+ 4975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:47.964820 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:47.964835 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:12:47.964869 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:47.968068 (+ 3199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:47.968870 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:47.968879 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:47.968885 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:47.969082 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:47.969458 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:47.969461 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:47.969463 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:47.969994 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:47.970277 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:47.970762 (+ 485us) server_negotiation.cc:299] Negotiation successful
0925 15:12:47.971193 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":379,"threads_started":1}
I20260925 15:12:47.974972 12929 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f5813fa68024a9b9509f46024db9a41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f5813fa68024a9b9509f46024db9a41" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 40783 } }
I20260925 15:12:47.975440 12929 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f5813fa68024a9b9509f46024db9a41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:12:47.975541 12929 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f5813fa68024a9b9509f46024db9a41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f5813fa68024a9b9509f46024db9a41, State: Initialized, Role: FOLLOWER
I20260925 15:12:47.976089 12929 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f5813fa68024a9b9509f46024db9a41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f5813fa68024a9b9509f46024db9a41" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 40783 } }
I20260925 15:12:47.976290 12929 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f5813fa68024a9b9509f46024db9a41 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:12:47.976435 12929 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f5813fa68024a9b9509f46024db9a41 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:12:47.976605 12929 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f5813fa68024a9b9509f46024db9a41 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:12:47.978747 12929 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f5813fa68024a9b9509f46024db9a41 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f5813fa68024a9b9509f46024db9a41" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 40783 } }
I20260925 15:12:47.979243 12929 leader_election.cc:302] T 00000000000000000000000000000000 P 6f5813fa68024a9b9509f46024db9a41 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f5813fa68024a9b9509f46024db9a41; no voters:
I20260925 15:12:47.979786 12929 leader_election.cc:288] T 00000000000000000000000000000000 P 6f5813fa68024a9b9509f46024db9a41 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:12:47.979938 12934 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f5813fa68024a9b9509f46024db9a41 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:12:47.980639 12934 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f5813fa68024a9b9509f46024db9a41 [term 1 LEADER]: Becoming Leader. State: Replica: 6f5813fa68024a9b9509f46024db9a41, State: Running, Role: LEADER
I20260925 15:12:47.981137 12934 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f5813fa68024a9b9509f46024db9a41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f5813fa68024a9b9509f46024db9a41" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 40783 } }
I20260925 15:12:47.981775 12929 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f5813fa68024a9b9509f46024db9a41 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:12:47.983975 12935 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f5813fa68024a9b9509f46024db9a41 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6f5813fa68024a9b9509f46024db9a41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f5813fa68024a9b9509f46024db9a41" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 40783 } } }
I20260925 15:12:47.984406 12935 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f5813fa68024a9b9509f46024db9a41 [sys.catalog]: This master's current role is: LEADER
I20260925 15:12:47.985811 12936 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f5813fa68024a9b9509f46024db9a41 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f5813fa68024a9b9509f46024db9a41. Latest consensus state: current_term: 1 leader_uuid: "6f5813fa68024a9b9509f46024db9a41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f5813fa68024a9b9509f46024db9a41" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 40783 } } }
I20260925 15:12:47.986279 12936 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f5813fa68024a9b9509f46024db9a41 [sys.catalog]: This master's current role is: LEADER
I20260925 15:12:47.987318 12941 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:12:47.997490 12941 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:12:48.007197 12941 catalog_manager.cc:1347] Generated new cluster ID: 4c8eae95d4144102bb522d46f8630b42
I20260925 15:12:48.007350 12941 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:12:48.032826 12941 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:12:48.034380 12941 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:12:48.042255 12941 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6f5813fa68024a9b9509f46024db9a41: Generated new TSK 0
I20260925 15:12:48.043143 12941 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.8.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.1.keytab.
Entry for principal kudu/127.12.8.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.1.keytab.
I20260925 15:12:48.093492 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349162370577-12320-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349162370577-12320-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349162370577-12320-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349162370577-12320-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.1:0
--local_ip_for_outbound_sockets=127.12.8.1
--webserver_interface=127.12.8.1
--webserver_port=0
--tserver_master_addrs=127.12.8.62:40783
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.1.keytab
--principal=kudu/127.12.8.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:40187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:12:48.405402 12955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:12:48.405674 12955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:12:48.405828 12955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:12:48.412920 12955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:12:48.413031 12955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:12:48.413213 12955 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.1
I20260925 15:12:48.422526 12955 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:40187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349162370577-12320-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349162370577-12320-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.1.keytab
--principal=kudu/127.12.8.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349162370577-12320-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.8.1
--webserver_port=0
--tserver_master_addrs=127.12.8.62:40783
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.1
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349162370577-12320-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:12:48.423751 12955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:12:48.425101 12955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:12:48.438006 12960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:12:48.437983 12961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:12:48.437942 12963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:12:48.438874 12955 server_base.cc:1034] running on GCE node
I20260925 15:12:48.439477 12955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:12:48.440337 12955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:12:48.441533 12955 hybrid_clock.cc:648] HybridClock initialized: now 1790349168441487 us; error 44 us; skew 500 ppm
I20260925 15:12:48.441823 12955 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:12:48 dist-test-slave-p468 krb5kdc[12849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349168, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:12:48.444866 12955 init.cc:377] Logged in from keytab as kudu/127.12.8.1@KRBTEST.COM (short username kudu)
I20260925 15:12:48.446367 12955 webserver.cc:460] Webserver started at http://127.12.8.1:44257/ using document root <none> and password file <none>
I20260925 15:12:48.447019 12955 fs_manager.cc:359] Metadata directory not provided
I20260925 15:12:48.447156 12955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:12:48.447449 12955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:12:48.450273 12955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349162370577-12320-0/minicluster-data/ts-0/data/instance:
uuid: "844392d0b1c14a20a9b9e4e533e03baf"
format_stamp: "Formatted at 2026-09-25 15:12:48 on dist-test-slave-p468"
server_key: "3bf540cc8986fdf1e369dd0ba3ce99a8"
server_key_iv: "6a20020d343b7fd23229b6b1615eda3b"
server_key_version: "encryptionkey@0"
I20260925 15:12:48.451049 12955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349162370577-12320-0/minicluster-data/ts-0/wal/instance:
uuid: "844392d0b1c14a20a9b9e4e533e03baf"
format_stamp: "Formatted at 2026-09-25 15:12:48 on dist-test-slave-p468"
server_key: "3bf540cc8986fdf1e369dd0ba3ce99a8"
server_key_iv: "6a20020d343b7fd23229b6b1615eda3b"
server_key_version: "encryptionkey@0"
I20260925 15:12:48.455760 12955 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 15:12:48.459018 12970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:12:48.460186 12955 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:12:48.460350 12955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349162370577-12320-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349162370577-12320-0/minicluster-data/ts-0/wal
uuid: "844392d0b1c14a20a9b9e4e533e03baf"
format_stamp: "Formatted at 2026-09-25 15:12:48 on dist-test-slave-p468"
server_key: "3bf540cc8986fdf1e369dd0ba3ce99a8"
server_key_iv: "6a20020d343b7fd23229b6b1615eda3b"
server_key_version: "encryptionkey@0"
I20260925 15:12:48.460549 12955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349162370577-12320-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349162370577-12320-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349162370577-12320-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:12:48.477510 12955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:12:48.481825 12955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:12:48.482100 12955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:12:48.483126 12955 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:12:48.484364 12955 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:12:48.484465 12955 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:12:48.484607 12955 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:12:48.484689 12955 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:12:48.517278 12955 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.1:38909
I20260925 15:12:48.517318 13082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.1:38909 every 8 connection(s)
I20260925 15:12:48.518551 12955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349162370577-12320-0/minicluster-data/ts-0/data/info.pb
I20260925 15:12:48.519718 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 12955
I20260925 15:12:48.519932 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349162370577-12320-0/minicluster-data/ts-0/wal/instance
I20260925 15:12:48.520526 12320 external_mini_cluster.cc:1428] Setting key 11df6ae6a3acd7dbc943f72189e4b382
Sep 25 15:12:48 dist-test-slave-p468 krb5kdc[12849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349168, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.1@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.8.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.2@KRBTEST.COM" created.
I20260925 15:12:48.540120 13085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:48.521125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.1:42317 (local address 127.12.8.62:40783)
0925 15:12:48.521754 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:48.521758 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:48.522788 (+ 1030us) server_negotiation.cc:407] Connection header received
0925 15:12:48.525608 (+ 2820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:48.525611 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:48.525671 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:48.525765 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:48.527869 (+ 2104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:48.528595 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:48.530106 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:48.530569 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:48.530703 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:48.534081 (+ 3378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:48.534086 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:48.534089 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:12:48.534118 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:48.536690 (+ 2572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:48.537598 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:48.537605 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:48.537608 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:48.537816 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:48.538292 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:48.538295 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:48.538297 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:48.538667 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:48.538779 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:48.539548 (+ 769us) server_negotiation.cc:299] Negotiation successful
0925 15:12:48.539719 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":1249,"threads_started":1}
I20260925 15:12:48.541016 13086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:48.521772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:40783 (local address 127.12.8.1:42317)
0925 15:12:48.522619 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:48.522669 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 15:12:48.525318 (+ 2649us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:12:48.526057 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:12:48.526070 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:12:48.526873 (+ 803us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:12:48.527681 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:48.527712 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:48.528747 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:48.528752 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:48.529955 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:48.529966 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:48.530798 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:48.530808 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:48.531040 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:48.531710 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:12:48.531734 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:12:48.533854 (+ 2120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:12:48.536833 (+ 2979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:12:48.536845 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:12:48.536863 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:12:48.537452 (+ 589us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:12:48.537953 (+ 501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:12:48.537957 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:12:48.537959 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:12:48.538165 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:12:48.538775 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:12:48.538780 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:12:48.539080 (+ 300us) client_negotiation.cc:769] Sending connection context
0925 15:12:48.539297 (+ 217us) client_negotiation.cc:240] Negotiation successful
0925 15:12:48.539659 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":315,"threads_started":1}
I20260925 15:12:48.542936 13083 heartbeater.cc:347] Connected to a master server at 127.12.8.62:40783
I20260925 15:12:48.543310 13083 heartbeater.cc:464] Registering TS with master...
I20260925 15:12:48.544133 13083 heartbeater.cc:511] Master 127.12.8.62:40783 requested a full tablet report, sending...
I20260925 15:12:48.546388 12894 ts_manager.cc:194] Registered new tserver with Master: 844392d0b1c14a20a9b9e4e533e03baf (127.12.8.1:38909)
I20260925 15:12:48.547771 12894 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.1@KRBTEST.COM'} at 127.12.8.1:42317
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.2.keytab.
Entry for principal kudu/127.12.8.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.2.keytab.
I20260925 15:12:48.561956 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349162370577-12320-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349162370577-12320-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349162370577-12320-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349162370577-12320-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.2:0
--local_ip_for_outbound_sockets=127.12.8.2
--webserver_interface=127.12.8.2
--webserver_port=0
--tserver_master_addrs=127.12.8.62:40783
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.2.keytab
--principal=kudu/127.12.8.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:40187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:12:48.885422 13089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:12:48.885756 13089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:12:48.885967 13089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:12:48.893678 13089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:12:48.893834 13089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:12:48.894007 13089 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.2
I20260925 15:12:48.903698 13089 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:40187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349162370577-12320-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349162370577-12320-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.2.keytab
--principal=kudu/127.12.8.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349162370577-12320-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.8.2
--webserver_port=0
--tserver_master_addrs=127.12.8.62:40783
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.2
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349162370577-12320-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:12:48.905023 13089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:12:48.906442 13089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:12:48.919562 13094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:12:48.919646 13095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:12:48.919600 13097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:12:48.920534 13089 server_base.cc:1034] running on GCE node
I20260925 15:12:48.921170 13089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:12:48.922086 13089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:12:48.923372 13089 hybrid_clock.cc:648] HybridClock initialized: now 1790349168923325 us; error 74 us; skew 500 ppm
I20260925 15:12:48.923738 13089 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:12:48 dist-test-slave-p468 krb5kdc[12849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349168, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:12:48.927199 13089 init.cc:377] Logged in from keytab as kudu/127.12.8.2@KRBTEST.COM (short username kudu)
I20260925 15:12:48.928683 13089 webserver.cc:460] Webserver started at http://127.12.8.2:40751/ using document root <none> and password file <none>
I20260925 15:12:48.929417 13089 fs_manager.cc:359] Metadata directory not provided
I20260925 15:12:48.929572 13089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:12:48.929929 13089 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:12:48.933007 13089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349162370577-12320-0/minicluster-data/ts-1/data/instance:
uuid: "adcc2e63361c49079378bba4968c869c"
format_stamp: "Formatted at 2026-09-25 15:12:48 on dist-test-slave-p468"
server_key: "0d2d811649856635c85074147c4234ee"
server_key_iv: "fe44f4cc253dbc8ccd8f33ea97cef208"
server_key_version: "encryptionkey@0"
I20260925 15:12:48.933823 13089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349162370577-12320-0/minicluster-data/ts-1/wal/instance:
uuid: "adcc2e63361c49079378bba4968c869c"
format_stamp: "Formatted at 2026-09-25 15:12:48 on dist-test-slave-p468"
server_key: "0d2d811649856635c85074147c4234ee"
server_key_iv: "fe44f4cc253dbc8ccd8f33ea97cef208"
server_key_version: "encryptionkey@0"
I20260925 15:12:48.939630 13089 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260925 15:12:48.943537 13104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:12:48.945178 13089 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 15:12:48.945366 13089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349162370577-12320-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349162370577-12320-0/minicluster-data/ts-1/wal
uuid: "adcc2e63361c49079378bba4968c869c"
format_stamp: "Formatted at 2026-09-25 15:12:48 on dist-test-slave-p468"
server_key: "0d2d811649856635c85074147c4234ee"
server_key_iv: "fe44f4cc253dbc8ccd8f33ea97cef208"
server_key_version: "encryptionkey@0"
I20260925 15:12:48.945580 13089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349162370577-12320-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349162370577-12320-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349162370577-12320-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:12:48.971117 13089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:12:48.975098 13089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:12:48.975325 13089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:12:48.976226 13089 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:12:48.977480 13089 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:12:48.977555 13089 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:12:48.977640 13089 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:12:48.977682 13089 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:12:49.012392 13089 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.2:36589
I20260925 15:12:49.012439 13216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.2:36589 every 8 connection(s)
I20260925 15:12:49.013715 13089 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349162370577-12320-0/minicluster-data/ts-1/data/info.pb
I20260925 15:12:49.017973 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 13089
I20260925 15:12:49.018184 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349162370577-12320-0/minicluster-data/ts-1/wal/instance
I20260925 15:12:49.018738 12320 external_mini_cluster.cc:1428] Setting key 2707ab3c63af4c1fe27a5e3e56681ec4
Sep 25 15:12:49 dist-test-slave-p468 krb5kdc[12849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349168, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.2@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
I20260925 15:12:49.036499 13085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:49.016291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.2:46793 (local address 127.12.8.62:40783)
0925 15:12:49.016469 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:49.016473 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:49.017705 (+ 1232us) server_negotiation.cc:407] Connection header received
0925 15:12:49.020021 (+ 2316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:49.020026 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:49.020101 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:49.020210 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:49.021991 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:49.022806 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:49.025263 (+ 2457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:49.025691 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:49.025784 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:49.029364 (+ 3580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:49.029373 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:49.029382 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:12:49.029423 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:49.033178 (+ 3755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:49.033836 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:49.033844 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:49.033851 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:49.034020 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:49.034561 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:49.034567 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:49.034572 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:49.034920 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:49.035080 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:49.035797 (+ 717us) server_negotiation.cc:299] Negotiation successful
0925 15:12:49.036062 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
WARNING: no policy specified for kudu/127.12.8.3@KRBTEST.COM; defaulting to no policy
I20260925 15:12:49.037240 13219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:49.016784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:40783 (local address 127.12.8.2:46793)
0925 15:12:49.017475 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:49.017518 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 15:12:49.019772 (+ 2254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:12:49.020411 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:12:49.020419 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:12:49.021026 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:12:49.021815 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:49.021841 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:49.022964 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:49.022973 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:49.025132 (+ 2159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:49.025149 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:49.025915 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:49.025922 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:49.026308 (+ 386us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:49.027012 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:12:49.027090 (+ 78us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:12:49.029146 (+ 2056us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:12:49.033308 (+ 4162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:12:49.033314 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:12:49.033327 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:12:49.033716 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:12:49.034165 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:12:49.034169 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:12:49.034172 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:12:49.034438 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:12:49.035068 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:12:49.035074 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:12:49.035419 (+ 345us) client_negotiation.cc:769] Sending connection context
0925 15:12:49.035651 (+ 232us) client_negotiation.cc:240] Negotiation successful
0925 15:12:49.036097 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":283,"threads_started":1}
I20260925 15:12:49.038815 13217 heartbeater.cc:347] Connected to a master server at 127.12.8.62:40783
I20260925 15:12:49.039103 13217 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.3@KRBTEST.COM" created.
I20260925 15:12:49.039853 13217 heartbeater.cc:511] Master 127.12.8.62:40783 requested a full tablet report, sending...
I20260925 15:12:49.041410 12894 ts_manager.cc:194] Registered new tserver with Master: adcc2e63361c49079378bba4968c869c (127.12.8.2:36589)
I20260925 15:12:49.042722 12894 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.2@KRBTEST.COM'} at 127.12.8.2:46793
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.3.keytab.
Entry for principal kudu/127.12.8.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.3.keytab.
I20260925 15:12:49.061053 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349162370577-12320-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349162370577-12320-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349162370577-12320-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349162370577-12320-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.3:0
--local_ip_for_outbound_sockets=127.12.8.3
--webserver_interface=127.12.8.3
--webserver_port=0
--tserver_master_addrs=127.12.8.62:40783
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.3.keytab
--principal=kudu/127.12.8.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:40187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:12:49.370661 13222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:12:49.370932 13222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:12:49.371026 13222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:12:49.378068 13222 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:12:49.378168 13222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:12:49.378342 13222 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.3
I20260925 15:12:49.388104 13222 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:40187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349162370577-12320-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349162370577-12320-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.3.keytab
--principal=kudu/127.12.8.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349162370577-12320-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.8.3
--webserver_port=0
--tserver_master_addrs=127.12.8.62:40783
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.3
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349162370577-12320-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:12:49.389287 13222 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:12:49.390586 13222 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:12:49.403187 13228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:12:49.403266 13227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:12:49.403182 13230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:12:49.404263 13222 server_base.cc:1034] running on GCE node
I20260925 15:12:49.404826 13222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:12:49.405843 13222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:12:49.407112 13222 hybrid_clock.cc:648] HybridClock initialized: now 1790349169407060 us; error 44 us; skew 500 ppm
I20260925 15:12:49.407449 13222 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:12:49 dist-test-slave-p468 krb5kdc[12849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349169, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:12:49.410565 13222 init.cc:377] Logged in from keytab as kudu/127.12.8.3@KRBTEST.COM (short username kudu)
I20260925 15:12:49.411995 13222 webserver.cc:460] Webserver started at http://127.12.8.3:34891/ using document root <none> and password file <none>
I20260925 15:12:49.412644 13222 fs_manager.cc:359] Metadata directory not provided
I20260925 15:12:49.412784 13222 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:12:49.413120 13222 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:12:49.416065 13222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349162370577-12320-0/minicluster-data/ts-2/data/instance:
uuid: "917cf46bd237492b9c89bd4de9714cfd"
format_stamp: "Formatted at 2026-09-25 15:12:49 on dist-test-slave-p468"
server_key: "52cc9940e6e7d4c7e9b75424b1b83f72"
server_key_iv: "389b4303069900a4cec4a19399212c0a"
server_key_version: "encryptionkey@0"
I20260925 15:12:49.416841 13222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349162370577-12320-0/minicluster-data/ts-2/wal/instance:
uuid: "917cf46bd237492b9c89bd4de9714cfd"
format_stamp: "Formatted at 2026-09-25 15:12:49 on dist-test-slave-p468"
server_key: "52cc9940e6e7d4c7e9b75424b1b83f72"
server_key_iv: "389b4303069900a4cec4a19399212c0a"
server_key_version: "encryptionkey@0"
I20260925 15:12:49.421721 13222 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260925 15:12:49.425304 13237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:12:49.426656 13222 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:12:49.426867 13222 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349162370577-12320-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349162370577-12320-0/minicluster-data/ts-2/wal
uuid: "917cf46bd237492b9c89bd4de9714cfd"
format_stamp: "Formatted at 2026-09-25 15:12:49 on dist-test-slave-p468"
server_key: "52cc9940e6e7d4c7e9b75424b1b83f72"
server_key_iv: "389b4303069900a4cec4a19399212c0a"
server_key_version: "encryptionkey@0"
I20260925 15:12:49.427062 13222 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349162370577-12320-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349162370577-12320-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349162370577-12320-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:12:49.445024 13222 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:12:49.449019 13222 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:12:49.449309 13222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:12:49.450299 13222 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:12:49.451632 13222 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:12:49.451701 13222 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:12:49.451784 13222 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:12:49.451839 13222 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:12:49.485970 13222 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.3:44685
I20260925 15:12:49.485991 13349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.3:44685 every 8 connection(s)
I20260925 15:12:49.487304 13222 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349162370577-12320-0/minicluster-data/ts-2/data/info.pb
I20260925 15:12:49.496739 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 13222
I20260925 15:12:49.496982 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349162370577-12320-0/minicluster-data/ts-2/wal/instance
I20260925 15:12:49.497509 12320 external_mini_cluster.cc:1428] Setting key 78e6b36acccdfeedc39d7e0e9b921558
Sep 25 15:12:49 dist-test-slave-p468 krb5kdc[12849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349169, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.3@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
I20260925 15:12:49.506850 13085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:49.489836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.3:34845 (local address 127.12.8.62:40783)
0925 15:12:49.490042 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:49.490048 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:49.491311 (+ 1263us) server_negotiation.cc:407] Connection header received
0925 15:12:49.493010 (+ 1699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:49.493015 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:49.493077 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:49.493165 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:49.494657 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:49.495451 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:49.497000 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:49.497598 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:49.497724 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:49.501225 (+ 3501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:49.501230 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:49.501233 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:12:49.501262 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:49.504101 (+ 2839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:49.504745 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:49.504749 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:49.504751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:49.504864 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:49.505245 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:49.505248 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:49.505250 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:49.505516 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:49.505624 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:49.506458 (+ 834us) server_negotiation.cc:299] Negotiation successful
0925 15:12:49.506619 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 15:12:49.507824 13352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:49.490462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:40783 (local address 127.12.8.3:34845)
0925 15:12:49.491151 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:49.491190 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 15:12:49.492751 (+ 1561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:12:49.493357 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:12:49.493365 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:12:49.493844 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:12:49.494491 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:49.494509 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:49.495586 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:49.495590 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:49.496837 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:49.496848 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:49.497785 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:49.497790 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:49.498015 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:49.498773 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:12:49.498810 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:12:49.501051 (+ 2241us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:12:49.504227 (+ 3176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:12:49.504233 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:12:49.504245 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:12:49.504633 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:12:49.504986 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:12:49.504989 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:12:49.504991 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:12:49.505152 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:12:49.505669 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:12:49.505675 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:12:49.506042 (+ 367us) client_negotiation.cc:769] Sending connection context
0925 15:12:49.506352 (+ 310us) client_negotiation.cc:240] Negotiation successful
0925 15:12:49.506723 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":258,"threads_started":1}
I20260925 15:12:49.509176 13350 heartbeater.cc:347] Connected to a master server at 127.12.8.62:40783
I20260925 15:12:49.509477 13350 heartbeater.cc:464] Registering TS with master...
I20260925 15:12:49.510161 13350 heartbeater.cc:511] Master 127.12.8.62:40783 requested a full tablet report, sending...
I20260925 15:12:49.511641 12894 ts_manager.cc:194] Registered new tserver with Master: 917cf46bd237492b9c89bd4de9714cfd (127.12.8.3:44685)
I20260925 15:12:49.512600 12320 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:12:49.512605 12894 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.3@KRBTEST.COM'} at 127.12.8.3:34845
I20260925 15:12:49.531213 13085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:49.518557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42190 (local address 127.12.8.62:40783)
0925 15:12:49.518702 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:49.518706 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:49.519244 (+ 538us) server_negotiation.cc:407] Connection header received
0925 15:12:49.519441 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:49.519446 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:49.519508 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:49.519607 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:49.521124 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:49.521805 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:49.523116 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:49.523526 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:49.523656 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:49.525903 (+ 2247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:49.525908 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:49.525911 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:12:49.525938 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:49.528493 (+ 2555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:49.529249 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:49.529256 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:49.529259 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:49.529390 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:49.529749 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:49.529756 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:49.529760 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:49.530106 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:49.530388 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:49.530719 (+ 331us) server_negotiation.cc:299] Negotiation successful
0925 15:12:49.530921 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 15:12:49.536999 12894 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42190:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:12:49.543944 12894 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:12:49.551326 13083 heartbeater.cc:503] Master 127.12.8.62:40783 was elected leader, sending a full tablet report...
I20260925 15:12:49.578809 13363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:49.555775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48546 (local address 127.12.8.3:44685)
0925 15:12:49.570099 (+ 14324us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:49.570111 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:49.570144 (+ 33us) server_negotiation.cc:407] Connection header received
0925 15:12:49.570266 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:49.570276 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:49.570542 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:49.570737 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:12:49.571964 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:49.573331 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:49.575845 (+ 2514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:49.577085 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:49.577437 (+ 352us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:49.577674 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:49.578296 (+ 622us) server_negotiation.cc:299] Negotiation successful
0925 15:12:49.578463 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14156,"thread_start_us":13997,"threads_started":1}
I20260925 15:12:49.579087 13364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:49.555961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40514 (local address 127.12.8.2:36589)
0925 15:12:49.568373 (+ 12412us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:49.568397 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:49.570654 (+ 2257us) server_negotiation.cc:407] Connection header received
0925 15:12:49.570758 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:49.570765 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:49.570990 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:49.571184 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:12:49.572146 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:49.573655 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:49.576390 (+ 2735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:49.577768 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:49.577887 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:49.578064 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:49.578598 (+ 534us) server_negotiation.cc:299] Negotiation successful
0925 15:12:49.578764 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12172,"thread_start_us":14957,"threads_started":1}
I20260925 15:12:49.579133 13362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:49.555910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.2:36589 (local address 127.0.0.1:40514)
0925 15:12:49.570066 (+ 14156us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:49.570084 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 15:12:49.570221 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:12:49.571416 (+ 1195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:12:49.571421 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:12:49.571466 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:12:49.571964 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:49.571975 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:49.573829 (+ 1854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:49.573835 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:49.576224 (+ 2389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:49.576238 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:49.578226 (+ 1988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:49.578233 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:49.578402 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:49.578435 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 15:12:49.578529 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 15:12:49.578718 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14044,"mutex_wait_us":5,"spinlock_wait_cycles":2816,"thread_start_us":12695,"threads_started":1}
I20260925 15:12:49.579466 13366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:49.569942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37862 (local address 127.12.8.1:38909)
0925 15:12:49.570541 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:49.570549 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:49.571708 (+ 1159us) server_negotiation.cc:407] Connection header received
0925 15:12:49.571918 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:49.571925 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:49.572119 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:49.572310 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:12:49.572839 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:49.574476 (+ 1637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:49.576829 (+ 2353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:49.578120 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:49.578268 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:49.578467 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:49.579003 (+ 536us) server_negotiation.cc:299] Negotiation successful
0925 15:12:49.579174 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":329,"threads_started":1}
I20260925 15:12:49.580130 13365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:49.555594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.3:44685 (local address 127.0.0.1:48546)
0925 15:12:49.569076 (+ 13482us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:49.569130 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 15:12:49.569393 (+ 263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:12:49.571186 (+ 1793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:12:49.571195 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:12:49.571227 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:12:49.571724 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:49.571745 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:49.573555 (+ 1810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:49.573560 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:49.575674 (+ 2114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:49.575685 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:49.577812 (+ 2127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:49.577822 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:49.578013 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:49.578060 (+ 47us) client_negotiation.cc:769] Sending connection context
0925 15:12:49.579423 (+ 1363us) client_negotiation.cc:240] Negotiation successful
0925 15:12:49.579675 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13158,"mutex_wait_us":108,"thread_start_us":14692,"threads_started":1}
I20260925 15:12:49.581058 13367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:49.570577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.1:38909 (local address 127.0.0.1:37862)
0925 15:12:49.571582 (+ 1005us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:49.571596 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 15:12:49.571754 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:12:49.572311 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:12:49.572315 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:12:49.572337 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:12:49.572664 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:49.572676 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:49.574648 (+ 1972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:49.574653 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:49.576691 (+ 2038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:49.576700 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:49.578629 (+ 1929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:49.578635 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:49.578794 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:49.578823 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 15:12:49.579704 (+ 881us) client_negotiation.cc:240] Negotiation successful
0925 15:12:49.579797 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":856,"mutex_wait_us":1,"thread_start_us":760,"threads_started":1}
I20260925 15:12:49.583384 13285 tablet_service.cc:1467] Processing CreateTablet for tablet 449c29a1f36b4b5fabf21c8d4ee2b316 (DEFAULT_TABLE table=test-table [id=5769e23545cb4281aad7dd8450c33685]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:12:49.583230 13152 tablet_service.cc:1467] Processing CreateTablet for tablet 449c29a1f36b4b5fabf21c8d4ee2b316 (DEFAULT_TABLE table=test-table [id=5769e23545cb4281aad7dd8450c33685]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:12:49.584537 13285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 449c29a1f36b4b5fabf21c8d4ee2b316. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:12:49.584613 13152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 449c29a1f36b4b5fabf21c8d4ee2b316. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:12:49.584300 13018 tablet_service.cc:1467] Processing CreateTablet for tablet 449c29a1f36b4b5fabf21c8d4ee2b316 (DEFAULT_TABLE table=test-table [id=5769e23545cb4281aad7dd8450c33685]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:12:49.585497 13018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 449c29a1f36b4b5fabf21c8d4ee2b316. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:12:49.595463 13368 tablet_bootstrap.cc:492] T 449c29a1f36b4b5fabf21c8d4ee2b316 P 917cf46bd237492b9c89bd4de9714cfd: Bootstrap starting.
I20260925 15:12:49.598583 13369 tablet_bootstrap.cc:492] T 449c29a1f36b4b5fabf21c8d4ee2b316 P adcc2e63361c49079378bba4968c869c: Bootstrap starting.
I20260925 15:12:49.599349 13368 tablet_bootstrap.cc:654] T 449c29a1f36b4b5fabf21c8d4ee2b316 P 917cf46bd237492b9c89bd4de9714cfd: Neither blocks nor log segments found. Creating new log.
I20260925 15:12:49.599901 13370 tablet_bootstrap.cc:492] T 449c29a1f36b4b5fabf21c8d4ee2b316 P 844392d0b1c14a20a9b9e4e533e03baf: Bootstrap starting.
I20260925 15:12:49.600736 13368 log.cc:824] T 449c29a1f36b4b5fabf21c8d4ee2b316 P 917cf46bd237492b9c89bd4de9714cfd: Log is configured to *not* fsync() on all Append() calls
I20260925 15:12:49.601857 13369 tablet_bootstrap.cc:654] T 449c29a1f36b4b5fabf21c8d4ee2b316 P adcc2e63361c49079378bba4968c869c: Neither blocks nor log segments found. Creating new log.
I20260925 15:12:49.603087 13370 tablet_bootstrap.cc:654] T 449c29a1f36b4b5fabf21c8d4ee2b316 P 844392d0b1c14a20a9b9e4e533e03baf: Neither blocks nor log segments found. Creating new log.
I20260925 15:12:49.603052 13369 log.cc:824] T 449c29a1f36b4b5fabf21c8d4ee2b316 P adcc2e63361c49079378bba4968c869c: Log is configured to *not* fsync() on all Append() calls
I20260925 15:12:49.603915 13368 tablet_bootstrap.cc:492] T 449c29a1f36b4b5fabf21c8d4ee2b316 P 917cf46bd237492b9c89bd4de9714cfd: No bootstrap required, opened a new log
I20260925 15:12:49.604187 13368 ts_tablet_manager.cc:1397] T 449c29a1f36b4b5fabf21c8d4ee2b316 P 917cf46bd237492b9c89bd4de9714cfd: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260925 15:12:49.604179 13370 log.cc:824] T 449c29a1f36b4b5fabf21c8d4ee2b316 P 844392d0b1c14a20a9b9e4e533e03baf: Log is configured to *not* fsync() on all Append() calls
I20260925 15:12:49.606374 13369 tablet_bootstrap.cc:492] T 449c29a1f36b4b5fabf21c8d4ee2b316 P adcc2e63361c49079378bba4968c869c: No bootstrap required, opened a new log
I20260925 15:12:49.606690 13369 ts_tablet_manager.cc:1397] T 449c29a1f36b4b5fabf21c8d4ee2b316 P adcc2e63361c49079378bba4968c869c: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.003s
I20260925 15:12:49.611428 13368 raft_consensus.cc:348] T 449c29a1f36b4b5fabf21c8d4ee2b316 P 917cf46bd237492b9c89bd4de9714cfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "917cf46bd237492b9c89bd4de9714cfd" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 44685 } } peers { permanent_uuid: "adcc2e63361c49079378bba4968c869c" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 36589 } } peers { permanent_uuid: "844392d0b1c14a20a9b9e4e533e03baf" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 38909 } }
I20260925 15:12:49.611773 13368 raft_consensus.cc:374] T 449c29a1f36b4b5fabf21c8d4ee2b316 P 917cf46bd237492b9c89bd4de9714cfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:12:49.611868 13368 raft_consensus.cc:729] T 449c29a1f36b4b5fabf21c8d4ee2b316 P 917cf46bd237492b9c89bd4de9714cfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 917cf46bd237492b9c89bd4de9714cfd, State: Initialized, Role: FOLLOWER
I20260925 15:12:49.612624 13370 tablet_bootstrap.cc:492] T 449c29a1f36b4b5fabf21c8d4ee2b316 P 844392d0b1c14a20a9b9e4e533e03baf: No bootstrap required, opened a new log
I20260925 15:12:49.612604 13368 consensus_queue.cc:260] T 449c29a1f36b4b5fabf21c8d4ee2b316 P 917cf46bd237492b9c89bd4de9714cfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "917cf46bd237492b9c89bd4de9714cfd" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 44685 } } peers { permanent_uuid: "adcc2e63361c49079378bba4968c869c" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 36589 } } peers { permanent_uuid: "844392d0b1c14a20a9b9e4e533e03baf" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 38909 } }
I20260925 15:12:49.612874 13370 ts_tablet_manager.cc:1397] T 449c29a1f36b4b5fabf21c8d4ee2b316 P 844392d0b1c14a20a9b9e4e533e03baf: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.008s
I20260925 15:12:49.614501 13350 heartbeater.cc:503] Master 127.12.8.62:40783 was elected leader, sending a full tablet report...
I20260925 15:12:49.615059 13368 ts_tablet_manager.cc:1428] T 449c29a1f36b4b5fabf21c8d4ee2b316 P 917cf46bd237492b9c89bd4de9714cfd: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260925 15:12:49.617012 13369 raft_consensus.cc:348] T 449c29a1f36b4b5fabf21c8d4ee2b316 P adcc2e63361c49079378bba4968c869c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "917cf46bd237492b9c89bd4de9714cfd" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 44685 } } peers { permanent_uuid: "adcc2e63361c49079378bba4968c869c" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 36589 } } peers { permanent_uuid: "844392d0b1c14a20a9b9e4e533e03baf" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 38909 } }
I20260925 15:12:49.617367 13369 raft_consensus.cc:374] T 449c29a1f36b4b5fabf21c8d4ee2b316 P adcc2e63361c49079378bba4968c869c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:12:49.617496 13369 raft_consensus.cc:729] T 449c29a1f36b4b5fabf21c8d4ee2b316 P adcc2e63361c49079378bba4968c869c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adcc2e63361c49079378bba4968c869c, State: Initialized, Role: FOLLOWER
I20260925 15:12:49.618078 13369 consensus_queue.cc:260] T 449c29a1f36b4b5fabf21c8d4ee2b316 P adcc2e63361c49079378bba4968c869c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "917cf46bd237492b9c89bd4de9714cfd" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 44685 } } peers { permanent_uuid: "adcc2e63361c49079378bba4968c869c" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 36589 } } peers { permanent_uuid: "844392d0b1c14a20a9b9e4e533e03baf" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 38909 } }
I20260925 15:12:49.620056 13217 heartbeater.cc:503] Master 127.12.8.62:40783 was elected leader, sending a full tablet report...
I20260925 15:12:49.620707 13369 ts_tablet_manager.cc:1428] T 449c29a1f36b4b5fabf21c8d4ee2b316 P adcc2e63361c49079378bba4968c869c: Time spent starting tablet: real 0.014s user 0.013s sys 0.001s
I20260925 15:12:49.622984 13370 raft_consensus.cc:348] T 449c29a1f36b4b5fabf21c8d4ee2b316 P 844392d0b1c14a20a9b9e4e533e03baf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "917cf46bd237492b9c89bd4de9714cfd" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 44685 } } peers { permanent_uuid: "adcc2e63361c49079378bba4968c869c" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 36589 } } peers { permanent_uuid: "844392d0b1c14a20a9b9e4e533e03baf" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 38909 } }
I20260925 15:12:49.623451 13370 raft_consensus.cc:374] T 449c29a1f36b4b5fabf21c8d4ee2b316 P 844392d0b1c14a20a9b9e4e533e03baf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:12:49.623557 13370 raft_consensus.cc:729] T 449c29a1f36b4b5fabf21c8d4ee2b316 P 844392d0b1c14a20a9b9e4e533e03baf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 844392d0b1c14a20a9b9e4e533e03baf, State: Initialized, Role: FOLLOWER
I20260925 15:12:49.624174 13370 consensus_queue.cc:260] T 449c29a1f36b4b5fabf21c8d4ee2b316 P 844392d0b1c14a20a9b9e4e533e03baf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "917cf46bd237492b9c89bd4de9714cfd" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 44685 } } peers { permanent_uuid: "adcc2e63361c49079378bba4968c869c" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 36589 } } peers { permanent_uuid: "844392d0b1c14a20a9b9e4e533e03baf" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 38909 } }
I20260925 15:12:49.625792 13370 ts_tablet_manager.cc:1428] T 449c29a1f36b4b5fabf21c8d4ee2b316 P 844392d0b1c14a20a9b9e4e533e03baf: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
W20260925 15:12:49.738960 13351 tablet.cc:2369] T 449c29a1f36b4b5fabf21c8d4ee2b316 P 917cf46bd237492b9c89bd4de9714cfd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:12:49.765722 13218 tablet.cc:2369] T 449c29a1f36b4b5fabf21c8d4ee2b316 P adcc2e63361c49079378bba4968c869c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:12:49.769253 13374 raft_consensus.cc:482] T 449c29a1f36b4b5fabf21c8d4ee2b316 P 917cf46bd237492b9c89bd4de9714cfd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:12:49.769560 13374 raft_consensus.cc:504] T 449c29a1f36b4b5fabf21c8d4ee2b316 P 917cf46bd237492b9c89bd4de9714cfd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "917cf46bd237492b9c89bd4de9714cfd" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 44685 } } peers { permanent_uuid: "adcc2e63361c49079378bba4968c869c" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 36589 } } peers { permanent_uuid: "844392d0b1c14a20a9b9e4e533e03baf" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 38909 } }
W20260925 15:12:49.770830 13084 tablet.cc:2369] T 449c29a1f36b4b5fabf21c8d4ee2b316 P 844392d0b1c14a20a9b9e4e533e03baf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:12:49.771606 13374 leader_election.cc:288] T 449c29a1f36b4b5fabf21c8d4ee2b316 P 917cf46bd237492b9c89bd4de9714cfd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers adcc2e63361c49079378bba4968c869c (127.12.8.2:36589), 844392d0b1c14a20a9b9e4e533e03baf (127.12.8.1:38909)
I20260925 15:12:49.778096 13364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:49.771904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.3:41153 (local address 127.12.8.2:36589)
0925 15:12:49.772083 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:49.772087 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:49.772297 (+ 210us) server_negotiation.cc:407] Connection header received
0925 15:12:49.772507 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:49.772513 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:49.772611 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:49.772718 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:12:49.773314 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:49.774391 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:49.776198 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:49.777301 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:49.777399 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:49.777474 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:49.777822 (+ 348us) server_negotiation.cc:299] Negotiation successful
0925 15:12:49.777909 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 15:12:49.778332 13352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:49.771953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.2:36589 (local address 127.12.8.3:41153)
0925 15:12:49.772176 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:49.772196 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 15:12:49.772348 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:12:49.772905 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:12:49.772910 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:12:49.772941 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:12:49.773135 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:49.773144 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:49.774526 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:49.774532 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:49.776040 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:49.776063 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:49.777455 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:49.777461 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:49.777663 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:49.777698 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 15:12:49.777805 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 15:12:49.777935 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260925 15:12:49.779045 13366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:49.772729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.3:50681 (local address 127.12.8.1:38909)
0925 15:12:49.772972 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:49.772980 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:49.773224 (+ 244us) server_negotiation.cc:407] Connection header received
0925 15:12:49.773377 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:49.773381 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:49.773439 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:49.773507 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:12:49.774088 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:49.775079 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:49.777054 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:49.778141 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:49.778292 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:49.778402 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:49.778740 (+ 338us) server_negotiation.cc:299] Negotiation successful
0925 15:12:49.778877 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260925 15:12:49.779114 13377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:49.772463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.1:38909 (local address 127.12.8.3:50681)
0925 15:12:49.773129 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:49.773142 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:12:49.773263 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:12:49.773549 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:12:49.773555 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:12:49.773580 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:12:49.773905 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:49.773915 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:49.775219 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:49.775222 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:49.776889 (+ 1667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:49.776904 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:49.778428 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:49.778459 (+ 31us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:49.778600 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:49.778623 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 15:12:49.778696 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 15:12:49.778772 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":381,"threads_started":1}
I20260925 15:12:49.779407 13172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "449c29a1f36b4b5fabf21c8d4ee2b316" candidate_uuid: "917cf46bd237492b9c89bd4de9714cfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adcc2e63361c49079378bba4968c869c" is_pre_election: true
I20260925 15:12:49.779979 13172 raft_consensus.cc:2454] T 449c29a1f36b4b5fabf21c8d4ee2b316 P adcc2e63361c49079378bba4968c869c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 917cf46bd237492b9c89bd4de9714cfd in term 0.
I20260925 15:12:49.780103 13038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "449c29a1f36b4b5fabf21c8d4ee2b316" candidate_uuid: "917cf46bd237492b9c89bd4de9714cfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "844392d0b1c14a20a9b9e4e533e03baf" is_pre_election: true
I20260925 15:12:49.780560 13038 raft_consensus.cc:2454] T 449c29a1f36b4b5fabf21c8d4ee2b316 P 844392d0b1c14a20a9b9e4e533e03baf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 917cf46bd237492b9c89bd4de9714cfd in term 0.
I20260925 15:12:49.780596 13239 leader_election.cc:302] T 449c29a1f36b4b5fabf21c8d4ee2b316 P 917cf46bd237492b9c89bd4de9714cfd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 917cf46bd237492b9c89bd4de9714cfd, adcc2e63361c49079378bba4968c869c; no voters:
I20260925 15:12:49.781014 13374 raft_consensus.cc:2781] T 449c29a1f36b4b5fabf21c8d4ee2b316 P 917cf46bd237492b9c89bd4de9714cfd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:12:49.781159 13374 raft_consensus.cc:482] T 449c29a1f36b4b5fabf21c8d4ee2b316 P 917cf46bd237492b9c89bd4de9714cfd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:12:49.781256 13374 raft_consensus.cc:3037] T 449c29a1f36b4b5fabf21c8d4ee2b316 P 917cf46bd237492b9c89bd4de9714cfd [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:12:49.783514 13374 raft_consensus.cc:504] T 449c29a1f36b4b5fabf21c8d4ee2b316 P 917cf46bd237492b9c89bd4de9714cfd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "917cf46bd237492b9c89bd4de9714cfd" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 44685 } } peers { permanent_uuid: "adcc2e63361c49079378bba4968c869c" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 36589 } } peers { permanent_uuid: "844392d0b1c14a20a9b9e4e533e03baf" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 38909 } }
I20260925 15:12:49.784073 13374 leader_election.cc:288] T 449c29a1f36b4b5fabf21c8d4ee2b316 P 917cf46bd237492b9c89bd4de9714cfd [CANDIDATE]: Term 1 election: Requested vote from peers adcc2e63361c49079378bba4968c869c (127.12.8.2:36589), 844392d0b1c14a20a9b9e4e533e03baf (127.12.8.1:38909)
I20260925 15:12:49.784621 13038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "449c29a1f36b4b5fabf21c8d4ee2b316" candidate_uuid: "917cf46bd237492b9c89bd4de9714cfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "844392d0b1c14a20a9b9e4e533e03baf"
I20260925 15:12:49.784653 13172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "449c29a1f36b4b5fabf21c8d4ee2b316" candidate_uuid: "917cf46bd237492b9c89bd4de9714cfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adcc2e63361c49079378bba4968c869c"
I20260925 15:12:49.784796 13038 raft_consensus.cc:3037] T 449c29a1f36b4b5fabf21c8d4ee2b316 P 844392d0b1c14a20a9b9e4e533e03baf [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:12:49.784835 13172 raft_consensus.cc:3037] T 449c29a1f36b4b5fabf21c8d4ee2b316 P adcc2e63361c49079378bba4968c869c [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:12:49.787099 13172 raft_consensus.cc:2454] T 449c29a1f36b4b5fabf21c8d4ee2b316 P adcc2e63361c49079378bba4968c869c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 917cf46bd237492b9c89bd4de9714cfd in term 1.
I20260925 15:12:49.787112 13038 raft_consensus.cc:2454] T 449c29a1f36b4b5fabf21c8d4ee2b316 P 844392d0b1c14a20a9b9e4e533e03baf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 917cf46bd237492b9c89bd4de9714cfd in term 1.
I20260925 15:12:49.787463 13239 leader_election.cc:302] T 449c29a1f36b4b5fabf21c8d4ee2b316 P 917cf46bd237492b9c89bd4de9714cfd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 917cf46bd237492b9c89bd4de9714cfd, adcc2e63361c49079378bba4968c869c; no voters:
I20260925 15:12:49.787786 13374 raft_consensus.cc:2781] T 449c29a1f36b4b5fabf21c8d4ee2b316 P 917cf46bd237492b9c89bd4de9714cfd [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:12:49.788233 13374 raft_consensus.cc:686] T 449c29a1f36b4b5fabf21c8d4ee2b316 P 917cf46bd237492b9c89bd4de9714cfd [term 1 LEADER]: Becoming Leader. State: Replica: 917cf46bd237492b9c89bd4de9714cfd, State: Running, Role: LEADER
I20260925 15:12:49.788673 13374 consensus_queue.cc:237] T 449c29a1f36b4b5fabf21c8d4ee2b316 P 917cf46bd237492b9c89bd4de9714cfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "917cf46bd237492b9c89bd4de9714cfd" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 44685 } } peers { permanent_uuid: "adcc2e63361c49079378bba4968c869c" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 36589 } } peers { permanent_uuid: "844392d0b1c14a20a9b9e4e533e03baf" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 38909 } }
I20260925 15:12:49.793161 12894 catalog_manager.cc:5495] T 449c29a1f36b4b5fabf21c8d4ee2b316 P 917cf46bd237492b9c89bd4de9714cfd reported cstate change: term changed from 0 to 1, leader changed from <none> to 917cf46bd237492b9c89bd4de9714cfd (127.12.8.3). New cstate: current_term: 1 leader_uuid: "917cf46bd237492b9c89bd4de9714cfd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "917cf46bd237492b9c89bd4de9714cfd" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 44685 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "adcc2e63361c49079378bba4968c869c" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 36589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "844392d0b1c14a20a9b9e4e533e03baf" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 38909 } health_report { overall_health: UNKNOWN } } }
Sep 25 15:12:49 dist-test-slave-p468 krb5kdc[12849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.1@KRBTEST.COM
I20260925 15:12:49.827090 13366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:49.812544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37874 (local address 127.12.8.1:38909)
0925 15:12:49.812733 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:49.812740 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:49.813192 (+ 452us) server_negotiation.cc:407] Connection header received
0925 15:12:49.813399 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:49.813403 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:49.813481 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:49.813592 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:49.815128 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:49.815889 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:49.817203 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:49.817628 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:49.817763 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:49.821817 (+ 4054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:49.821831 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:49.821844 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:12:49.821880 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:49.824803 (+ 2923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:49.825396 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:49.825402 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:49.825404 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:49.825518 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:49.825824 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:49.825827 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:49.825829 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:49.826176 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:49.826464 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:49.826691 (+ 227us) server_negotiation.cc:299] Negotiation successful
0925 15:12:49.826838 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
Sep 25 15:12:49 dist-test-slave-p468 krb5kdc[12849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349169, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 15:12:49 dist-test-slave-p468 krb5kdc[12849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349169, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.8.1@KRBTEST.COM
I20260925 15:12:49.881294 13366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:49.865303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37886 (local address 127.12.8.1:38909)
0925 15:12:49.865588 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:49.865595 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:49.866151 (+ 556us) server_negotiation.cc:407] Connection header received
0925 15:12:49.866428 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:49.866438 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:49.866553 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:49.866696 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:49.868391 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:49.869444 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:49.871298 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:49.872053 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:49.872158 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:49.876304 (+ 4146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:49.876309 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:49.876312 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:12:49.876346 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:49.878869 (+ 2523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:49.879449 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:49.879452 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:49.879454 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:49.879573 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:49.879978 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:49.879982 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:49.879985 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:49.880243 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:49.880485 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:49.880749 (+ 264us) server_negotiation.cc:299] Negotiation successful
0925 15:12:49.880904 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144}
Sep 25 15:12:49 dist-test-slave-p468 krb5kdc[12849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349169, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 15:12:49 dist-test-slave-p468 krb5kdc[12849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349169, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.1@KRBTEST.COM
I20260925 15:12:49.926031 13366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:49.912632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37890 (local address 127.12.8.1:38909)
0925 15:12:49.912865 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:49.912872 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:49.913028 (+ 156us) server_negotiation.cc:407] Connection header received
0925 15:12:49.913190 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:49.913194 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:49.913257 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:49.913346 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:49.914709 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:49.915534 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:49.916756 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:49.917176 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:49.917268 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:49.920946 (+ 3678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:49.920952 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:49.920955 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:12:49.920993 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:49.923528 (+ 2535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:49.924146 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:49.924153 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:49.924156 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:49.924278 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:49.924657 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:49.924662 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:49.924667 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:49.924955 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:49.925211 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:49.925477 (+ 266us) server_negotiation.cc:299] Negotiation successful
0925 15:12:49.925697 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
W20260925 15:12:49.927743 12998 tablet_service.cc:2908] Rejecting scan request for tablet 449c29a1f36b4b5fabf21c8d4ee2b316: Uninitialized: clean time has not yet been initialized
I20260925 15:12:49.930752 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 12955
I20260925 15:12:49.943475 13366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:49.931596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37900 (local address 127.12.8.1:38909)
0925 15:12:49.931770 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:49.931774 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:49.932173 (+ 399us) server_negotiation.cc:407] Connection header received
0925 15:12:49.932260 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:49.932264 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:49.932331 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:49.932406 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:49.933424 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:49.934085 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:49.935308 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:49.935713 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:49.935862 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:49.937819 (+ 1957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:49.937852 (+ 33us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:49.937856 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:12:49.937882 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:49.940903 (+ 3021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:49.941492 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:49.941497 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:49.941501 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:49.941629 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:49.942154 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:49.942159 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:49.942163 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:49.942503 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:49.942777 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:49.942955 (+ 178us) server_negotiation.cc:299] Negotiation successful
0925 15:12:49.943154 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 15:12:49.944104 13078 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:12:50.035322 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 12955
I20260925 15:12:50.052338 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 13089
Sep 25 15:12:50 dist-test-slave-p468 krb5kdc[12849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349169, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.2@KRBTEST.COM
I20260925 15:12:50.066715 13364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:50.053167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40522 (local address 127.12.8.2:36589)
0925 15:12:50.053348 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:50.053354 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:50.053374 (+ 20us) server_negotiation.cc:407] Connection header received
0925 15:12:50.053474 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:50.053487 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:50.053576 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:50.053685 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:50.054916 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:50.055675 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:50.056947 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:50.057333 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:50.057466 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:50.061271 (+ 3805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:50.061284 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:50.061299 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:12:50.061333 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:50.064194 (+ 2861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:50.064842 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:50.064849 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:50.064851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:50.064982 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:50.065324 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:50.065327 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:50.065328 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:50.065665 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:50.065865 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:50.066298 (+ 433us) server_negotiation.cc:299] Negotiation successful
0925 15:12:50.066469 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 15:12:50.067482 13212 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:12:50.154858 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 13089
I20260925 15:12:50.171350 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 13222
I20260925 15:12:50.178766 13377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:50.178392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.1:38909 (local address 127.12.8.3:44081)
0925 15:12:50.178540 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:50.178639 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.1:38909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260925 15:12:50.179105 13239 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.8.1:38909: connect: Connection refused (error 111)
I20260925 15:12:50.180620 13377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:50.180299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.1:38909 (local address 127.12.8.3:48383)
0925 15:12:50.180442 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:50.180504 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.1:38909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
Sep 25 15:12:50 dist-test-slave-p468 krb5kdc[12849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349169, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.3@KRBTEST.COM
W20260925 15:12:50.181371 13239 consensus_peers.cc:487] T 449c29a1f36b4b5fabf21c8d4ee2b316 P 917cf46bd237492b9c89bd4de9714cfd -> Peer 844392d0b1c14a20a9b9e4e533e03baf (127.12.8.1:38909): Couldn't send request to peer 844392d0b1c14a20a9b9e4e533e03baf. Status: Network error: Client connection negotiation failed: client connection to 127.12.8.1:38909: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 15:12:50.187996 13393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:50.172134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48556 (local address 127.12.8.3:44685)
0925 15:12:50.172625 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:50.172629 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:50.172644 (+ 15us) server_negotiation.cc:407] Connection header received
0925 15:12:50.172698 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:50.172701 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:50.172764 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:50.172856 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:50.174289 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:50.174976 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:50.176348 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:50.176726 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:50.176821 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:50.182342 (+ 5521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:50.182354 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:50.182368 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:12:50.182405 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:50.185237 (+ 2832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:50.185851 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:50.185864 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:50.185866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:50.186074 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:50.186520 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:50.186524 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:50.186526 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:50.186840 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:50.187105 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:50.187479 (+ 374us) server_negotiation.cc:299] Negotiation successful
0925 15:12:50.187632 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":280,"threads_started":1}
I20260925 15:12:50.188597 13345 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:12:50.276394 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 13222
I20260925 15:12:50.276486 13377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:50.276032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.2:36589 (local address 127.12.8.3:41359)
0925 15:12:50.276203 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:50.276310 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.2:36589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260925 15:12:50.292873 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 12862
I20260925 15:12:50.293980 12924 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:12:50.401365 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 12862
2026-09-25T15:12:50Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4130 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:12:50 dist-test-slave-p468 krb5kdc[13399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:12:50 dist-test-slave-p468 krb5kdc[13399](info): set up 2 sockets
Sep 25 15:12:50 dist-test-slave-p468 krb5kdc[13399](info): commencing operation
krb5kdc: starting...
W20260925 15:12:51.482231 12320 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.000s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/krb5kdc/test-user.keytab.
Sep 25 15:12:51 dist-test-slave-p468 krb5kdc[13399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:12:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:12:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.8.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab.
Entry for principal kudu/127.12.8.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab.
I20260925 15:12:51.657644 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.62:39285
--webserver_interface=127.12.8.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab
--principal=kudu/127.12.8.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:34305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.62:39285
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:12:51.969816 13412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:12:51.970086 13412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:12:51.970264 13412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:12:51.977286 13412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:12:51.977411 13412 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 15:12:51.977490 13412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:12:51.977598 13412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:12:51.977681 13412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:12:51.977766 13412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:12:51.987361 13412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:34305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.8.62:39285
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab
--principal=kudu/127.12.8.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.62:39285
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.8.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:12:51.988577 13412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:12:51.989914 13412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:12:52.001214 13417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:12:52.001344 13418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:12:52.001714 13420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:12:52.002128 13412 server_base.cc:1034] running on GCE node
I20260925 15:12:52.002897 13412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:12:52.004175 13412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:12:52.005441 13412 hybrid_clock.cc:648] HybridClock initialized: now 1790349172005340 us; error 39 us; skew 500 ppm
I20260925 15:12:52.005765 13412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:12:52 dist-test-slave-p468 krb5kdc[13399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349172, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:12:52.009145 13412 init.cc:377] Logged in from keytab as kudu/127.12.8.62@KRBTEST.COM (short username kudu)
I20260925 15:12:52.010628 13412 webserver.cc:460] Webserver started at http://127.12.8.62:40857/ using document root <none> and password file <none>
I20260925 15:12:52.011292 13412 fs_manager.cc:359] Metadata directory not provided
I20260925 15:12:52.011415 13412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:12:52.011729 13412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:12:52.014586 13412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/minicluster-data/master-0/data/instance:
uuid: "6676713ee26a4eb188da557ac038061e"
format_stamp: "Formatted at 2026-09-25 15:12:52 on dist-test-slave-p468"
server_key: "7599c6dd4a1c7ad271a10b70fc010f0d"
server_key_iv: "8e36ae8f729f037cf2f401334f70d870"
server_key_version: "encryptionkey@0"
I20260925 15:12:52.015326 13412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/minicluster-data/master-0/wal/instance:
uuid: "6676713ee26a4eb188da557ac038061e"
format_stamp: "Formatted at 2026-09-25 15:12:52 on dist-test-slave-p468"
server_key: "7599c6dd4a1c7ad271a10b70fc010f0d"
server_key_iv: "8e36ae8f729f037cf2f401334f70d870"
server_key_version: "encryptionkey@0"
I20260925 15:12:52.020360 13412 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 15:12:52.023766 13427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:12:52.025004 13412 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 15:12:52.025182 13412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/minicluster-data/master-0/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/minicluster-data/master-0/wal
uuid: "6676713ee26a4eb188da557ac038061e"
format_stamp: "Formatted at 2026-09-25 15:12:52 on dist-test-slave-p468"
server_key: "7599c6dd4a1c7ad271a10b70fc010f0d"
server_key_iv: "8e36ae8f729f037cf2f401334f70d870"
server_key_version: "encryptionkey@0"
I20260925 15:12:52.025398 13412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:12:52.040805 13412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:12:52.045027 13412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:12:52.045291 13412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:12:52.065645 13412 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.62:39285
I20260925 15:12:52.065665 13488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.62:39285 every 8 connection(s)
I20260925 15:12:52.066972 13412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
I20260925 15:12:52.070425 13489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:12:52.072794 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 13412
I20260925 15:12:52.073041 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/minicluster-data/master-0/wal/instance
I20260925 15:12:52.073819 12320 external_mini_cluster.cc:1428] Setting key 5fb3ecf7603650f85b8b215ad62b2527
I20260925 15:12:52.081722 13489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6676713ee26a4eb188da557ac038061e: Bootstrap starting.
I20260925 15:12:52.084619 13489 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6676713ee26a4eb188da557ac038061e: Neither blocks nor log segments found. Creating new log.
I20260925 15:12:52.085582 13489 log.cc:824] T 00000000000000000000000000000000 P 6676713ee26a4eb188da557ac038061e: Log is configured to *not* fsync() on all Append() calls
Sep 25 15:12:52 dist-test-slave-p468 krb5kdc[13399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
I20260925 15:12:52.088480 13489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6676713ee26a4eb188da557ac038061e: No bootstrap required, opened a new log
I20260925 15:12:52.096140 13489 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6676713ee26a4eb188da557ac038061e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6676713ee26a4eb188da557ac038061e" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 39285 } }
I20260925 15:12:52.096539 13492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:52.075920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36932 (local address 127.12.8.62:39285)
0925 15:12:52.076650 (+ 730us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:52.076661 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:52.076699 (+ 38us) server_negotiation.cc:407] Connection header received
0925 15:12:52.078384 (+ 1685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:52.078404 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:52.078743 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:52.079142 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:52.080464 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:52.081932 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:52.083201 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:52.083843 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:52.084036 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:52.088514 (+ 4478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:52.088522 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:52.088568 (+ 46us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:12:52.088624 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:52.092110 (+ 3486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:52.092892 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:52.092899 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:52.092905 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:52.093078 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:52.093519 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:52.093530 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:52.093534 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:52.094123 (+ 589us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:52.094556 (+ 433us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:52.095012 (+ 456us) server_negotiation.cc:299] Negotiation successful
0925 15:12:52.095449 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":340,"threads_started":1}
I20260925 15:12:52.096518 13489 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6676713ee26a4eb188da557ac038061e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:12:52.096724 13489 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6676713ee26a4eb188da557ac038061e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6676713ee26a4eb188da557ac038061e, State: Initialized, Role: FOLLOWER
I20260925 15:12:52.097296 13489 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6676713ee26a4eb188da557ac038061e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6676713ee26a4eb188da557ac038061e" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 39285 } }
I20260925 15:12:52.097563 13489 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6676713ee26a4eb188da557ac038061e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:12:52.097709 13489 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6676713ee26a4eb188da557ac038061e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:12:52.097885 13489 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6676713ee26a4eb188da557ac038061e [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:12:52.100683 13489 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6676713ee26a4eb188da557ac038061e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6676713ee26a4eb188da557ac038061e" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 39285 } }
I20260925 15:12:52.101172 13489 leader_election.cc:302] T 00000000000000000000000000000000 P 6676713ee26a4eb188da557ac038061e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6676713ee26a4eb188da557ac038061e; no voters:
I20260925 15:12:52.101696 13489 leader_election.cc:288] T 00000000000000000000000000000000 P 6676713ee26a4eb188da557ac038061e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:12:52.101827 13494 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6676713ee26a4eb188da557ac038061e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:12:52.103288 13494 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6676713ee26a4eb188da557ac038061e [term 1 LEADER]: Becoming Leader. State: Replica: 6676713ee26a4eb188da557ac038061e, State: Running, Role: LEADER
I20260925 15:12:52.103799 13489 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6676713ee26a4eb188da557ac038061e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:12:52.103757 13494 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6676713ee26a4eb188da557ac038061e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6676713ee26a4eb188da557ac038061e" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 39285 } }
I20260925 15:12:52.106485 13496 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6676713ee26a4eb188da557ac038061e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6676713ee26a4eb188da557ac038061e. Latest consensus state: current_term: 1 leader_uuid: "6676713ee26a4eb188da557ac038061e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6676713ee26a4eb188da557ac038061e" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 39285 } } }
I20260925 15:12:52.106904 13496 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6676713ee26a4eb188da557ac038061e [sys.catalog]: This master's current role is: LEADER
I20260925 15:12:52.107506 13495 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6676713ee26a4eb188da557ac038061e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6676713ee26a4eb188da557ac038061e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6676713ee26a4eb188da557ac038061e" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 39285 } } }
I20260925 15:12:52.107827 13495 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6676713ee26a4eb188da557ac038061e [sys.catalog]: This master's current role is: LEADER
I20260925 15:12:52.108920 13502 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:12:52.113797 13502 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:12:52.123402 13502 catalog_manager.cc:1347] Generated new cluster ID: d488c9954b554e04beaa3a45b2e5b5b5
I20260925 15:12:52.123535 13502 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:12:52.146421 13502 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:12:52.147753 13502 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:12:52.156924 13502 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6676713ee26a4eb188da557ac038061e: Generated new TSK 0
I20260925 15:12:52.157722 13502 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.8.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.1.keytab.
Entry for principal kudu/127.12.8.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.1.keytab.
I20260925 15:12:52.199553 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.1:0
--local_ip_for_outbound_sockets=127.12.8.1
--webserver_interface=127.12.8.1
--webserver_port=0
--tserver_master_addrs=127.12.8.62:39285
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.1.keytab
--principal=kudu/127.12.8.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:34305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:12:52.525872 13515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:12:52.526129 13515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:12:52.526309 13515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:12:52.533540 13515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:12:52.533636 13515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:12:52.533715 13515 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 15:12:52.533800 13515 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.1
I20260925 15:12:52.543270 13515 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:34305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.1.keytab
--principal=kudu/127.12.8.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.8.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.8.62:39285
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.1
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:12:52.544391 13515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:12:52.545594 13515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:12:52.562558 13520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:12:52.562508 13523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:12:52.562477 13521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:12:52.563407 13515 server_base.cc:1034] running on GCE node
I20260925 15:12:52.564035 13515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:12:52.564932 13515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:12:52.566174 13515 hybrid_clock.cc:648] HybridClock initialized: now 1790349172566113 us; error 41 us; skew 500 ppm
I20260925 15:12:52.566517 13515 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:12:52 dist-test-slave-p468 krb5kdc[13399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349172, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:12:52.569506 13515 init.cc:377] Logged in from keytab as kudu/127.12.8.1@KRBTEST.COM (short username kudu)
I20260925 15:12:52.570974 13515 webserver.cc:460] Webserver started at http://127.12.8.1:33629/ using document root <none> and password file <none>
I20260925 15:12:52.571621 13515 fs_manager.cc:359] Metadata directory not provided
I20260925 15:12:52.571751 13515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:12:52.572080 13515 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:12:52.574951 13515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/minicluster-data/ts-0/data/instance:
uuid: "93a588223ed64f8abf3dc71226199d7c"
format_stamp: "Formatted at 2026-09-25 15:12:52 on dist-test-slave-p468"
server_key: "c8e6920e3633e9f4c5eefeeaef03054f"
server_key_iv: "849b29c60ed7ce8fbf061952048857d6"
server_key_version: "encryptionkey@0"
I20260925 15:12:52.575727 13515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/minicluster-data/ts-0/wal/instance:
uuid: "93a588223ed64f8abf3dc71226199d7c"
format_stamp: "Formatted at 2026-09-25 15:12:52 on dist-test-slave-p468"
server_key: "c8e6920e3633e9f4c5eefeeaef03054f"
server_key_iv: "849b29c60ed7ce8fbf061952048857d6"
server_key_version: "encryptionkey@0"
I20260925 15:12:52.580605 13515 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260925 15:12:52.583833 13530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:12:52.585004 13515 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 15:12:52.585177 13515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/minicluster-data/ts-0/wal
uuid: "93a588223ed64f8abf3dc71226199d7c"
format_stamp: "Formatted at 2026-09-25 15:12:52 on dist-test-slave-p468"
server_key: "c8e6920e3633e9f4c5eefeeaef03054f"
server_key_iv: "849b29c60ed7ce8fbf061952048857d6"
server_key_version: "encryptionkey@0"
I20260925 15:12:52.585402 13515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:12:52.595534 13515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:12:52.599301 13515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:12:52.599579 13515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:12:52.602066 13515 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:12:52.602198 13515 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:12:52.602370 13515 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:12:52.602448 13515 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 15:12:52 dist-test-slave-p468 krb5kdc[13399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349172, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.1@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
I20260925 15:12:52.623822 13538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:52.602406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.1:40905 (local address 127.12.8.62:39285)
0925 15:12:52.602908 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:52.602914 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:52.604225 (+ 1311us) server_negotiation.cc:407] Connection header received
0925 15:12:52.607203 (+ 2978us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:52.607209 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:52.607297 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:52.607439 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:52.609154 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:52.610431 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:52.612162 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:52.612935 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:52.613093 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:52.618002 (+ 4909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:52.618010 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:52.618017 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:12:52.618062 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:52.620869 (+ 2807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:52.621569 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:52.621581 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:52.621584 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:52.621696 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:52.622112 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:52.622119 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:52.622122 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:52.622444 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:52.622605 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:52.623241 (+ 636us) server_negotiation.cc:299] Negotiation successful
0925 15:12:52.623468 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":227,"threads_started":1}
I20260925 15:12:52.624791 13540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:52.603084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:39285 (local address 127.12.8.1:40905)
0925 15:12:52.604025 (+ 941us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:52.604097 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0925 15:12:52.606863 (+ 2766us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:12:52.607714 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:12:52.607727 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:12:52.608341 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:12:52.608937 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:52.608982 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:52.610609 (+ 1627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:52.610612 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:52.612032 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:52.612044 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:52.613199 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:52.613207 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:52.613419 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:52.614295 (+ 876us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:12:52.614333 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:12:52.617767 (+ 3434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:12:52.621012 (+ 3245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:12:52.621021 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:12:52.621036 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:12:52.621440 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:12:52.621804 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:12:52.621810 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:12:52.621813 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:12:52.622004 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:12:52.622582 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:12:52.622590 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:12:52.622913 (+ 323us) client_negotiation.cc:769] Sending connection context
0925 15:12:52.623171 (+ 258us) client_negotiation.cc:240] Negotiation successful
0925 15:12:52.623677 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":633,"thread_start_us":365,"threads_started":1}
I20260925 15:12:52.645567 13538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:52.630483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.1:34129 (local address 127.12.8.62:39285)
0925 15:12:52.630634 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:52.630638 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:52.631339 (+ 701us) server_negotiation.cc:407] Connection header received
0925 15:12:52.631546 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:52.631550 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:52.631603 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:52.631698 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:52.632558 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:52.633307 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:52.635897 (+ 2590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:52.636303 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:52.636396 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:52.639095 (+ 2699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:52.639101 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:52.639105 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:12:52.639146 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:52.642421 (+ 3275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:52.643359 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:52.643363 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:52.643365 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:52.643480 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:52.643929 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:52.643934 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:52.643937 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:52.644332 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:52.644524 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:52.644968 (+ 444us) server_negotiation.cc:299] Negotiation successful
0925 15:12:52.645189 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 15:12:52.645690 13583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:52.630607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:39285 (local address 127.12.8.1:34129)
0925 15:12:52.631239 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:52.631253 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 15:12:52.631402 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:12:52.631720 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:12:52.631724 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:12:52.632162 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:12:52.632423 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:52.632432 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:52.633497 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:52.633504 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:52.635728 (+ 2224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:52.635758 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:52.636514 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:52.636523 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:52.636801 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:52.637496 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:12:52.637526 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:12:52.638892 (+ 1366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:12:52.642553 (+ 3661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:12:52.642561 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:12:52.642568 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:12:52.643236 (+ 668us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:12:52.643601 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:12:52.643606 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:12:52.643610 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:12:52.643800 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:12:52.644483 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:12:52.644492 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:12:52.644717 (+ 225us) client_negotiation.cc:769] Sending connection context
0925 15:12:52.644869 (+ 152us) client_negotiation.cc:240] Negotiation successful
0925 15:12:52.645207 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":391,"threads_started":1}
I20260925 15:12:52.660282 13515 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.1:45655
I20260925 15:12:52.660316 13650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.1:45655 every 8 connection(s)
I20260925 15:12:52.661516 13515 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/minicluster-data/ts-0/data/info.pb
I20260925 15:12:52.663558 13651 heartbeater.cc:347] Connected to a master server at 127.12.8.62:39285
I20260925 15:12:52.663736 13651 heartbeater.cc:464] Registering TS with master...
I20260925 15:12:52.664289 13651 heartbeater.cc:511] Master 127.12.8.62:39285 requested a full tablet report, sending...
I20260925 15:12:52.666047 13444 ts_manager.cc:194] Registered new tserver with Master: 93a588223ed64f8abf3dc71226199d7c (127.12.8.1:45655)
I20260925 15:12:52.666137 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 13515
I20260925 15:12:52.666378 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/minicluster-data/ts-0/wal/instance
I20260925 15:12:52.666996 12320 external_mini_cluster.cc:1428] Setting key e2ccb8241c19c3deefc4d4c0c5292f65
I20260925 15:12:52.667703 13444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.1@KRBTEST.COM'} at 127.12.8.1:40905
WARNING: no policy specified for kudu/127.12.8.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.2.keytab.
Entry for principal kudu/127.12.8.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.2.keytab.
I20260925 15:12:52.705240 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.2:0
--local_ip_for_outbound_sockets=127.12.8.2
--webserver_interface=127.12.8.2
--webserver_port=0
--tserver_master_addrs=127.12.8.62:39285
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.2.keytab
--principal=kudu/127.12.8.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:34305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:12:53.029827 13655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:12:53.030112 13655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:12:53.030236 13655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:12:53.038727 13655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:12:53.038839 13655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:12:53.038923 13655 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 15:12:53.039001 13655 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.2
I20260925 15:12:53.049009 13655 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:34305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.2.keytab
--principal=kudu/127.12.8.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.8.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.8.62:39285
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.2
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:12:53.050402 13655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:12:53.051628 13655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:12:53.064003 13663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:12:53.064062 13660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:12:53.064082 13661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:12:53.064944 13655 server_base.cc:1034] running on GCE node
I20260925 15:12:53.065583 13655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:12:53.066622 13655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:12:53.067867 13655 hybrid_clock.cc:648] HybridClock initialized: now 1790349173067799 us; error 68 us; skew 500 ppm
I20260925 15:12:53.068181 13655 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:12:53 dist-test-slave-p468 krb5kdc[13399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349173, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:12:53.076521 13655 init.cc:377] Logged in from keytab as kudu/127.12.8.2@KRBTEST.COM (short username kudu)
I20260925 15:12:53.077940 13655 webserver.cc:460] Webserver started at http://127.12.8.2:42387/ using document root <none> and password file <none>
I20260925 15:12:53.078626 13655 fs_manager.cc:359] Metadata directory not provided
I20260925 15:12:53.078735 13655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:12:53.079068 13655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:12:53.081753 13655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/minicluster-data/ts-1/data/instance:
uuid: "0d1a68cc0c8c4273abdb41ff336583c6"
format_stamp: "Formatted at 2026-09-25 15:12:53 on dist-test-slave-p468"
server_key: "c4eabc80a5d4caab8971f8f68d26a822"
server_key_iv: "33a3a4f6ca81ebd50a8315a8113e8a25"
server_key_version: "encryptionkey@0"
I20260925 15:12:53.082580 13655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/minicluster-data/ts-1/wal/instance:
uuid: "0d1a68cc0c8c4273abdb41ff336583c6"
format_stamp: "Formatted at 2026-09-25 15:12:53 on dist-test-slave-p468"
server_key: "c4eabc80a5d4caab8971f8f68d26a822"
server_key_iv: "33a3a4f6ca81ebd50a8315a8113e8a25"
server_key_version: "encryptionkey@0"
I20260925 15:12:53.087703 13655 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 15:12:53.091203 13670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:12:53.092484 13655 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 15:12:53.092655 13655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/minicluster-data/ts-1/wal
uuid: "0d1a68cc0c8c4273abdb41ff336583c6"
format_stamp: "Formatted at 2026-09-25 15:12:53 on dist-test-slave-p468"
server_key: "c4eabc80a5d4caab8971f8f68d26a822"
server_key_iv: "33a3a4f6ca81ebd50a8315a8113e8a25"
server_key_version: "encryptionkey@0"
I20260925 15:12:53.092861 13655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:12:53.111366 13655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:12:53.115052 13655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:12:53.115322 13655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:12:53.117787 13655 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:12:53.117892 13655 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:12:53.118021 13655 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:12:53.118081 13655 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 15:12:53 dist-test-slave-p468 krb5kdc[13399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349173, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.2@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
I20260925 15:12:53.141024 13538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:53.118446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.2:35247 (local address 127.12.8.62:39285)
0925 15:12:53.118639 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:53.118645 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:53.120076 (+ 1431us) server_negotiation.cc:407] Connection header received
0925 15:12:53.122255 (+ 2179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:53.122262 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:53.122354 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:53.122481 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:53.124556 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:53.125656 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:53.128430 (+ 2774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:53.129023 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:53.129147 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:53.134077 (+ 4930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:53.134085 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:53.134091 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:12:53.134135 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:53.137546 (+ 3411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:53.138239 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:53.138247 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:53.138251 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:53.138386 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:53.138961 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:53.138969 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:53.138972 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:53.139335 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:53.139511 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:53.140446 (+ 935us) server_negotiation.cc:299] Negotiation successful
0925 15:12:53.140658 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 15:12:53.142050 13679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:53.119067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:39285 (local address 127.12.8.2:35247)
0925 15:12:53.119872 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:53.119926 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 15:12:53.121965 (+ 2039us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:12:53.122732 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:12:53.122749 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:12:53.123496 (+ 747us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:12:53.124383 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:53.124410 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:53.125828 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:53.125834 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:53.128269 (+ 2435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:53.128288 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:53.129460 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:53.129466 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:53.129665 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:53.130609 (+ 944us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:12:53.130645 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:12:53.133776 (+ 3131us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:12:53.137699 (+ 3923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:12:53.137707 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:12:53.137721 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:12:53.138101 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:12:53.138501 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:12:53.138506 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:12:53.138510 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:12:53.138815 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:12:53.139592 (+ 777us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:12:53.139597 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:12:53.140097 (+ 500us) client_negotiation.cc:769] Sending connection context
0925 15:12:53.140363 (+ 266us) client_negotiation.cc:240] Negotiation successful
0925 15:12:53.140870 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":344,"threads_started":1}
I20260925 15:12:53.161854 13538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:53.148319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.2:49021 (local address 127.12.8.62:39285)
0925 15:12:53.148508 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:53.148515 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:53.149367 (+ 852us) server_negotiation.cc:407] Connection header received
0925 15:12:53.149548 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:53.149554 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:53.149619 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:53.149715 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:53.150773 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:53.151522 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:53.153453 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:53.153860 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:53.153988 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:53.156463 (+ 2475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:53.156469 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:53.156475 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:12:53.156517 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:53.159199 (+ 2682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:53.159992 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:53.160000 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:53.160003 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:53.160113 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:53.160460 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:53.160465 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:53.160468 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:53.160764 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:53.160909 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:53.161286 (+ 377us) server_negotiation.cc:299] Negotiation successful
0925 15:12:53.161513 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 15:12:53.161965 13748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:53.148687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:39285 (local address 127.12.8.2:49021)
0925 15:12:53.149266 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:53.149280 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 15:12:53.149432 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:12:53.149735 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:12:53.149739 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:12:53.150157 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:12:53.150569 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:53.150578 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:53.151748 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:53.151756 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:53.153306 (+ 1550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:53.153320 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:53.154053 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:53.154064 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:53.154321 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:53.155097 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:12:53.155116 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:12:53.156274 (+ 1158us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:12:53.159321 (+ 3047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:12:53.159328 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:12:53.159333 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:12:53.159868 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:12:53.160224 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:12:53.160229 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:12:53.160233 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:12:53.160352 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:12:53.160875 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:12:53.160880 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:12:53.161069 (+ 189us) client_negotiation.cc:769] Sending connection context
0925 15:12:53.161204 (+ 135us) client_negotiation.cc:240] Negotiation successful
0925 15:12:53.161499 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":350,"threads_started":1}
I20260925 15:12:53.168883 13655 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.2:42187
I20260925 15:12:53.168905 13789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.2:42187 every 8 connection(s)
I20260925 15:12:53.170040 13655 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/minicluster-data/ts-1/data/info.pb
I20260925 15:12:53.171576 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 13655
I20260925 15:12:53.171770 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/minicluster-data/ts-1/wal/instance
I20260925 15:12:53.172272 12320 external_mini_cluster.cc:1428] Setting key eec096aa8ffee081a35bd2dca70c8208
I20260925 15:12:53.172078 13790 heartbeater.cc:347] Connected to a master server at 127.12.8.62:39285
I20260925 15:12:53.173166 13790 heartbeater.cc:464] Registering TS with master...
I20260925 15:12:53.173923 13790 heartbeater.cc:511] Master 127.12.8.62:39285 requested a full tablet report, sending...
I20260925 15:12:53.175241 13443 ts_manager.cc:194] Registered new tserver with Master: 0d1a68cc0c8c4273abdb41ff336583c6 (127.12.8.2:42187)
I20260925 15:12:53.176061 13443 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.2@KRBTEST.COM'} at 127.12.8.2:35247
WARNING: no policy specified for kudu/127.12.8.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.3.keytab.
Entry for principal kudu/127.12.8.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.3.keytab.
I20260925 15:12:53.214231 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.3:0
--local_ip_for_outbound_sockets=127.12.8.3
--webserver_interface=127.12.8.3
--webserver_port=0
--tserver_master_addrs=127.12.8.62:39285
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.3.keytab
--principal=kudu/127.12.8.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:34305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:12:53.529192 13794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:12:53.529503 13794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:12:53.529695 13794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:12:53.536646 13794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:12:53.536736 13794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:12:53.536907 13794 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 15:12:53.537037 13794 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.3
I20260925 15:12:53.546387 13794 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:34305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.3.keytab
--principal=kudu/127.12.8.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.8.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.8.62:39285
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.3
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:12:53.547542 13794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:12:53.548790 13794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:12:53.566505 13799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:12:53.566531 13800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:12:53.566461 13802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:12:53.567307 13794 server_base.cc:1034] running on GCE node
I20260925 15:12:53.567891 13794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:12:53.568740 13794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:12:53.569981 13794 hybrid_clock.cc:648] HybridClock initialized: now 1790349173569913 us; error 62 us; skew 500 ppm
I20260925 15:12:53.570327 13794 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:12:53 dist-test-slave-p468 krb5kdc[13399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349173, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:12:53.573388 13794 init.cc:377] Logged in from keytab as kudu/127.12.8.3@KRBTEST.COM (short username kudu)
I20260925 15:12:53.575014 13794 webserver.cc:460] Webserver started at http://127.12.8.3:37105/ using document root <none> and password file <none>
I20260925 15:12:53.575703 13794 fs_manager.cc:359] Metadata directory not provided
I20260925 15:12:53.575850 13794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:12:53.576174 13794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:12:53.579018 13794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/minicluster-data/ts-2/data/instance:
uuid: "f6720378ff514b54b38e689fd93699ef"
format_stamp: "Formatted at 2026-09-25 15:12:53 on dist-test-slave-p468"
server_key: "56101cce6c64ab22fcf041b65cf0f5ef"
server_key_iv: "a36b13320a269bd87202266516c5a236"
server_key_version: "encryptionkey@0"
I20260925 15:12:53.579787 13794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/minicluster-data/ts-2/wal/instance:
uuid: "f6720378ff514b54b38e689fd93699ef"
format_stamp: "Formatted at 2026-09-25 15:12:53 on dist-test-slave-p468"
server_key: "56101cce6c64ab22fcf041b65cf0f5ef"
server_key_iv: "a36b13320a269bd87202266516c5a236"
server_key_version: "encryptionkey@0"
I20260925 15:12:53.584858 13794 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 15:12:53.588284 13809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:12:53.589851 13794 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260925 15:12:53.590027 13794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/minicluster-data/ts-2/wal
uuid: "f6720378ff514b54b38e689fd93699ef"
format_stamp: "Formatted at 2026-09-25 15:12:53 on dist-test-slave-p468"
server_key: "56101cce6c64ab22fcf041b65cf0f5ef"
server_key_iv: "a36b13320a269bd87202266516c5a236"
server_key_version: "encryptionkey@0"
I20260925 15:12:53.590265 13794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:12:53.617241 13794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:12:53.621603 13794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:12:53.621821 13794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:12:53.624394 13794 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:12:53.624471 13794 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:12:53.624620 13794 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:12:53.624678 13794 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 15:12:53 dist-test-slave-p468 krb5kdc[13399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349173, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.3@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
I20260925 15:12:53.655517 13818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:53.625734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:39285 (local address 127.12.8.3:40757)
0925 15:12:53.626686 (+ 952us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:53.626729 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 15:12:53.628989 (+ 2260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:12:53.630091 (+ 1102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:12:53.630103 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:12:53.630954 (+ 851us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:12:53.632000 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:53.632033 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:53.633481 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:53.633488 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:53.635762 (+ 2274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:53.635774 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:53.636589 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:53.636596 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:53.636952 (+ 356us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:53.637723 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:12:53.637772 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:12:53.641037 (+ 3265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:12:53.650113 (+ 9076us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:12:53.650129 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:12:53.650173 (+ 44us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:12:53.651126 (+ 953us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:12:53.651559 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:12:53.651564 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:12:53.651568 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:12:53.651863 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:12:53.652602 (+ 739us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:12:53.652608 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:12:53.653139 (+ 531us) client_negotiation.cc:769] Sending connection context
0925 15:12:53.653506 (+ 367us) client_negotiation.cc:240] Negotiation successful
0925 15:12:53.654073 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":680,"thread_start_us":444,"threads_started":1}
I20260925 15:12:53.656776 13538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:53.625238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.3:40757 (local address 127.12.8.62:39285)
0925 15:12:53.625430 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:53.625439 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:53.626861 (+ 1422us) server_negotiation.cc:407] Connection header received
0925 15:12:53.629397 (+ 2536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:53.629402 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:53.629497 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:53.629609 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:53.632212 (+ 2603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:53.633326 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:53.635921 (+ 2595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:53.636349 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:53.636483 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:53.641325 (+ 4842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:53.641345 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:53.641353 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:12:53.641408 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:53.649954 (+ 8546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:53.651287 (+ 1333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:53.651295 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:53.651299 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:53.651434 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:53.652025 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:53.652028 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:53.652030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:53.652385 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:53.652531 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:53.656217 (+ 3686us) server_negotiation.cc:299] Negotiation successful
0925 15:12:53.656502 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 15:12:53.667160 13794 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.3:34991
I20260925 15:12:53.667213 13928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.3:34991 every 8 connection(s)
I20260925 15:12:53.668406 13794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/minicluster-data/ts-2/data/info.pb
I20260925 15:12:53.670461 13929 heartbeater.cc:347] Connected to a master server at 127.12.8.62:39285
I20260925 15:12:53.670673 13929 heartbeater.cc:464] Registering TS with master...
I20260925 15:12:53.670913 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 13794
I20260925 15:12:53.670969 13651 heartbeater.cc:503] Master 127.12.8.62:39285 was elected leader, sending a full tablet report...
I20260925 15:12:53.671095 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/minicluster-data/ts-2/wal/instance
I20260925 15:12:53.671423 13929 heartbeater.cc:511] Master 127.12.8.62:39285 requested a full tablet report, sending...
I20260925 15:12:53.671615 12320 external_mini_cluster.cc:1428] Setting key 7c3a36e4464e8108d6da6b9c76dadfc5
I20260925 15:12:53.672968 13444 ts_manager.cc:194] Registered new tserver with Master: f6720378ff514b54b38e689fd93699ef (127.12.8.3:34991)
I20260925 15:12:53.673709 13444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.3@KRBTEST.COM'} at 127.12.8.3:40757
I20260925 15:12:53.676344 12320 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:12:53.679330 13925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:53.664882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:39285 (local address 127.12.8.3:52431)
0925 15:12:53.665554 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:53.665578 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 15:12:53.665777 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:12:53.666368 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:12:53.666375 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:12:53.667021 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:12:53.667526 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:53.667538 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:53.668523 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:53.668528 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:53.670449 (+ 1921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:53.670463 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:53.671678 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:53.671686 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:53.671869 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:53.672563 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:12:53.672587 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:12:53.673340 (+ 753us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:12:53.677105 (+ 3765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:12:53.677109 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:12:53.677112 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:12:53.677469 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:12:53.677836 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:12:53.677839 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:12:53.677841 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:12:53.677952 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:12:53.678545 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:12:53.678551 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:12:53.678701 (+ 150us) client_negotiation.cc:769] Sending connection context
0925 15:12:53.678824 (+ 123us) client_negotiation.cc:240] Negotiation successful
0925 15:12:53.679043 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":370,"threads_started":1}
I20260925 15:12:53.679670 13538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:53.664870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.3:52431 (local address 127.12.8.62:39285)
0925 15:12:53.665049 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:53.665056 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:53.665668 (+ 612us) server_negotiation.cc:407] Connection header received
0925 15:12:53.665924 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:53.665929 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:53.666018 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:53.666102 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:53.667671 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:53.668388 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:53.670658 (+ 2270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:53.671291 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:53.671405 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:53.673508 (+ 2103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:53.673514 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:53.673519 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:12:53.673562 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:53.676987 (+ 3425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:53.677576 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:53.677581 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:53.677585 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:53.677733 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:53.678048 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:53.678053 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:53.678057 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:53.678396 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:53.678548 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:53.678973 (+ 425us) server_negotiation.cc:299] Negotiation successful
0925 15:12:53.679224 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
Sep 25 15:12:53 dist-test-slave-p468 krb5kdc[13399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349173, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 15:12:53 dist-test-slave-p468 krb5kdc[13399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349173, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
I20260925 15:12:53.721495 13538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:53.708011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38736 (local address 127.12.8.62:39285)
0925 15:12:53.708228 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:53.708235 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:53.708394 (+ 159us) server_negotiation.cc:407] Connection header received
0925 15:12:53.708544 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:53.708548 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:53.708620 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:53.708707 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:53.710156 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:53.710968 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:53.712244 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:53.712652 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:53.712800 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:53.716517 (+ 3717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:53.716522 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:53.716526 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:12:53.716558 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:53.719230 (+ 2672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:53.719792 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:53.719795 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:53.719798 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:53.719941 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:53.720247 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:53.720250 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:53.720252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:53.720517 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:53.720681 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:53.720955 (+ 274us) server_negotiation.cc:299] Negotiation successful
0925 15:12:53.721120 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 15:12:53.725672 13444 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38736:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:12:53.732867 13444 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:12:53.764038 13942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:53.744714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.1:45655 (local address 127.0.0.1:60428)
0925 15:12:53.745702 (+ 988us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:53.745744 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 15:12:53.745976 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:12:53.756361 (+ 10385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:12:53.756368 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:12:53.756405 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:12:53.756775 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:53.756788 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:53.758857 (+ 2069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:53.758862 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:53.761540 (+ 2678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:53.761554 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:53.763266 (+ 1712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:53.763271 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:53.763439 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:53.763467 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 15:12:53.763545 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 15:12:53.763641 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":698,"mutex_wait_us":104,"thread_start_us":1055,"threads_started":1}
I20260925 15:12:53.764060 13944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:53.744973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60428 (local address 127.12.8.1:45655)
0925 15:12:53.754856 (+ 9883us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:53.754866 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:53.754901 (+ 35us) server_negotiation.cc:407] Connection header received
0925 15:12:53.754997 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:53.755005 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:53.755320 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:53.757103 (+ 1783us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:12:53.757328 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:53.758656 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:53.761695 (+ 3039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:53.763079 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:53.763207 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:53.763401 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:53.763603 (+ 202us) server_negotiation.cc:299] Negotiation successful
0925 15:12:53.763756 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9711,"thread_start_us":9582,"threads_started":1}
I20260925 15:12:53.764536 13945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:53.745542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41780 (local address 127.12.8.3:34991)
0925 15:12:53.756382 (+ 10840us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:53.756390 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:53.756430 (+ 40us) server_negotiation.cc:407] Connection header received
0925 15:12:53.756526 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:53.756532 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:53.756687 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:53.756884 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:12:53.757932 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:53.759505 (+ 1573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:53.761407 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:53.762836 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:53.762973 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:53.763209 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:53.764049 (+ 840us) server_negotiation.cc:299] Negotiation successful
0925 15:12:53.764228 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10661,"thread_start_us":10478,"threads_started":1}
I20260925 15:12:53.764861 13946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:53.744919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.3:34991 (local address 127.0.0.1:41780)
0925 15:12:53.753988 (+ 9069us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:53.754018 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0925 15:12:53.754137 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:12:53.757191 (+ 3054us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:12:53.757196 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:12:53.757218 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:12:53.757716 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:53.757727 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:53.759637 (+ 1910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:53.759642 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:53.761247 (+ 1605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:53.761258 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:53.763010 (+ 1752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:53.763014 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:53.763170 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:53.763231 (+ 61us) client_negotiation.cc:769] Sending connection context
0925 15:12:53.763386 (+ 155us) client_negotiation.cc:240] Negotiation successful
0925 15:12:53.763584 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8919,"mutex_wait_us":5,"thread_start_us":10745,"threads_started":1}
I20260925 15:12:53.765908 13943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:53.745602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.2:42187 (local address 127.0.0.1:41926)
0925 15:12:53.755997 (+ 10395us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:53.756019 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 15:12:53.756129 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:12:53.756837 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:12:53.756841 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:12:53.756867 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:12:53.757298 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:53.757312 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:53.759145 (+ 1833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:53.759150 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:53.761540 (+ 2390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:53.761554 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:53.765144 (+ 3590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:53.765150 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:53.765298 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:53.765327 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 15:12:53.765435 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 15:12:53.765554 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10246,"mutex_wait_us":30,"thread_start_us":8045,"threads_started":1}
I20260925 15:12:53.766474 13947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:53.746309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41926 (local address 127.12.8.2:42187)
0925 15:12:53.754380 (+ 8071us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:53.754393 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:53.756247 (+ 1854us) server_negotiation.cc:407] Connection header received
0925 15:12:53.756359 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:53.756380 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:53.756617 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:53.756849 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:12:53.757503 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:53.759010 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:53.763298 (+ 4288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:53.764569 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:53.764697 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:53.764891 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:53.765482 (+ 591us) server_negotiation.cc:299] Negotiation successful
0925 15:12:53.765640 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7849,"thread_start_us":7704,"threads_started":1}
I20260925 15:12:53.768129 13586 tablet_service.cc:1467] Processing CreateTablet for tablet dc6e71699e744f73ba056c0359f722eb (DEFAULT_TABLE table=test-table [id=598259c1c82540ea8c690ca4ed9f50d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:12:53.768198 13859 tablet_service.cc:1467] Processing CreateTablet for tablet dc6e71699e744f73ba056c0359f722eb (DEFAULT_TABLE table=test-table [id=598259c1c82540ea8c690ca4ed9f50d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:12:53.769445 13586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc6e71699e744f73ba056c0359f722eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:12:53.769519 13859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc6e71699e744f73ba056c0359f722eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:12:53.769620 13720 tablet_service.cc:1467] Processing CreateTablet for tablet dc6e71699e744f73ba056c0359f722eb (DEFAULT_TABLE table=test-table [id=598259c1c82540ea8c690ca4ed9f50d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:12:53.770826 13720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc6e71699e744f73ba056c0359f722eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:12:53.780125 13948 tablet_bootstrap.cc:492] T dc6e71699e744f73ba056c0359f722eb P 93a588223ed64f8abf3dc71226199d7c: Bootstrap starting.
I20260925 15:12:53.783270 13948 tablet_bootstrap.cc:654] T dc6e71699e744f73ba056c0359f722eb P 93a588223ed64f8abf3dc71226199d7c: Neither blocks nor log segments found. Creating new log.
I20260925 15:12:53.783576 13949 tablet_bootstrap.cc:492] T dc6e71699e744f73ba056c0359f722eb P f6720378ff514b54b38e689fd93699ef: Bootstrap starting.
I20260925 15:12:53.787333 13949 tablet_bootstrap.cc:654] T dc6e71699e744f73ba056c0359f722eb P f6720378ff514b54b38e689fd93699ef: Neither blocks nor log segments found. Creating new log.
I20260925 15:12:53.788650 13949 log.cc:824] T dc6e71699e744f73ba056c0359f722eb P f6720378ff514b54b38e689fd93699ef: Log is configured to *not* fsync() on all Append() calls
I20260925 15:12:53.790587 13948 log.cc:824] T dc6e71699e744f73ba056c0359f722eb P 93a588223ed64f8abf3dc71226199d7c: Log is configured to *not* fsync() on all Append() calls
I20260925 15:12:53.793756 13948 tablet_bootstrap.cc:492] T dc6e71699e744f73ba056c0359f722eb P 93a588223ed64f8abf3dc71226199d7c: No bootstrap required, opened a new log
I20260925 15:12:53.794039 13948 ts_tablet_manager.cc:1397] T dc6e71699e744f73ba056c0359f722eb P 93a588223ed64f8abf3dc71226199d7c: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.006s
I20260925 15:12:53.794680 13950 tablet_bootstrap.cc:492] T dc6e71699e744f73ba056c0359f722eb P 0d1a68cc0c8c4273abdb41ff336583c6: Bootstrap starting.
I20260925 15:12:53.794802 13949 tablet_bootstrap.cc:492] T dc6e71699e744f73ba056c0359f722eb P f6720378ff514b54b38e689fd93699ef: No bootstrap required, opened a new log
I20260925 15:12:53.795097 13949 ts_tablet_manager.cc:1397] T dc6e71699e744f73ba056c0359f722eb P f6720378ff514b54b38e689fd93699ef: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260925 15:12:53.798074 13950 tablet_bootstrap.cc:654] T dc6e71699e744f73ba056c0359f722eb P 0d1a68cc0c8c4273abdb41ff336583c6: Neither blocks nor log segments found. Creating new log.
I20260925 15:12:53.799050 13950 log.cc:824] T dc6e71699e744f73ba056c0359f722eb P 0d1a68cc0c8c4273abdb41ff336583c6: Log is configured to *not* fsync() on all Append() calls
I20260925 15:12:53.802094 13950 tablet_bootstrap.cc:492] T dc6e71699e744f73ba056c0359f722eb P 0d1a68cc0c8c4273abdb41ff336583c6: No bootstrap required, opened a new log
I20260925 15:12:53.802390 13950 ts_tablet_manager.cc:1397] T dc6e71699e744f73ba056c0359f722eb P 0d1a68cc0c8c4273abdb41ff336583c6: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.004s
I20260925 15:12:53.805522 13948 raft_consensus.cc:348] T dc6e71699e744f73ba056c0359f722eb P 93a588223ed64f8abf3dc71226199d7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a588223ed64f8abf3dc71226199d7c" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 45655 } } peers { permanent_uuid: "f6720378ff514b54b38e689fd93699ef" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 34991 } } peers { permanent_uuid: "0d1a68cc0c8c4273abdb41ff336583c6" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 42187 } }
I20260925 15:12:53.805522 13949 raft_consensus.cc:348] T dc6e71699e744f73ba056c0359f722eb P f6720378ff514b54b38e689fd93699ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a588223ed64f8abf3dc71226199d7c" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 45655 } } peers { permanent_uuid: "f6720378ff514b54b38e689fd93699ef" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 34991 } } peers { permanent_uuid: "0d1a68cc0c8c4273abdb41ff336583c6" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 42187 } }
I20260925 15:12:53.805945 13948 raft_consensus.cc:374] T dc6e71699e744f73ba056c0359f722eb P 93a588223ed64f8abf3dc71226199d7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:12:53.805945 13949 raft_consensus.cc:374] T dc6e71699e744f73ba056c0359f722eb P f6720378ff514b54b38e689fd93699ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:12:53.806051 13949 raft_consensus.cc:729] T dc6e71699e744f73ba056c0359f722eb P f6720378ff514b54b38e689fd93699ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6720378ff514b54b38e689fd93699ef, State: Initialized, Role: FOLLOWER
I20260925 15:12:53.806084 13948 raft_consensus.cc:729] T dc6e71699e744f73ba056c0359f722eb P 93a588223ed64f8abf3dc71226199d7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93a588223ed64f8abf3dc71226199d7c, State: Initialized, Role: FOLLOWER
I20260925 15:12:53.806984 13949 consensus_queue.cc:260] T dc6e71699e744f73ba056c0359f722eb P f6720378ff514b54b38e689fd93699ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a588223ed64f8abf3dc71226199d7c" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 45655 } } peers { permanent_uuid: "f6720378ff514b54b38e689fd93699ef" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 34991 } } peers { permanent_uuid: "0d1a68cc0c8c4273abdb41ff336583c6" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 42187 } }
I20260925 15:12:53.807111 13948 consensus_queue.cc:260] T dc6e71699e744f73ba056c0359f722eb P 93a588223ed64f8abf3dc71226199d7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a588223ed64f8abf3dc71226199d7c" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 45655 } } peers { permanent_uuid: "f6720378ff514b54b38e689fd93699ef" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 34991 } } peers { permanent_uuid: "0d1a68cc0c8c4273abdb41ff336583c6" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 42187 } }
I20260925 15:12:53.808094 13929 heartbeater.cc:503] Master 127.12.8.62:39285 was elected leader, sending a full tablet report...
I20260925 15:12:53.809167 13948 ts_tablet_manager.cc:1428] T dc6e71699e744f73ba056c0359f722eb P 93a588223ed64f8abf3dc71226199d7c: Time spent starting tablet: real 0.015s user 0.012s sys 0.003s
I20260925 15:12:53.811580 13950 raft_consensus.cc:348] T dc6e71699e744f73ba056c0359f722eb P 0d1a68cc0c8c4273abdb41ff336583c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a588223ed64f8abf3dc71226199d7c" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 45655 } } peers { permanent_uuid: "f6720378ff514b54b38e689fd93699ef" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 34991 } } peers { permanent_uuid: "0d1a68cc0c8c4273abdb41ff336583c6" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 42187 } }
I20260925 15:12:53.812033 13950 raft_consensus.cc:374] T dc6e71699e744f73ba056c0359f722eb P 0d1a68cc0c8c4273abdb41ff336583c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:12:53.812180 13950 raft_consensus.cc:729] T dc6e71699e744f73ba056c0359f722eb P 0d1a68cc0c8c4273abdb41ff336583c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d1a68cc0c8c4273abdb41ff336583c6, State: Initialized, Role: FOLLOWER
I20260925 15:12:53.812382 13949 ts_tablet_manager.cc:1428] T dc6e71699e744f73ba056c0359f722eb P f6720378ff514b54b38e689fd93699ef: Time spent starting tablet: real 0.017s user 0.008s sys 0.006s
I20260925 15:12:53.812922 13950 consensus_queue.cc:260] T dc6e71699e744f73ba056c0359f722eb P 0d1a68cc0c8c4273abdb41ff336583c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a588223ed64f8abf3dc71226199d7c" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 45655 } } peers { permanent_uuid: "f6720378ff514b54b38e689fd93699ef" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 34991 } } peers { permanent_uuid: "0d1a68cc0c8c4273abdb41ff336583c6" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 42187 } }
I20260925 15:12:53.814860 13790 heartbeater.cc:503] Master 127.12.8.62:39285 was elected leader, sending a full tablet report...
I20260925 15:12:53.815490 13950 ts_tablet_manager.cc:1428] T dc6e71699e744f73ba056c0359f722eb P 0d1a68cc0c8c4273abdb41ff336583c6: Time spent starting tablet: real 0.013s user 0.006s sys 0.005s
I20260925 15:12:53.895679 13955 raft_consensus.cc:482] T dc6e71699e744f73ba056c0359f722eb P 93a588223ed64f8abf3dc71226199d7c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:12:53.896088 13955 raft_consensus.cc:504] T dc6e71699e744f73ba056c0359f722eb P 93a588223ed64f8abf3dc71226199d7c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a588223ed64f8abf3dc71226199d7c" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 45655 } } peers { permanent_uuid: "f6720378ff514b54b38e689fd93699ef" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 34991 } } peers { permanent_uuid: "0d1a68cc0c8c4273abdb41ff336583c6" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 42187 } }
I20260925 15:12:53.897744 13955 leader_election.cc:288] T dc6e71699e744f73ba056c0359f722eb P 93a588223ed64f8abf3dc71226199d7c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6720378ff514b54b38e689fd93699ef (127.12.8.3:34991), 0d1a68cc0c8c4273abdb41ff336583c6 (127.12.8.2:42187)
I20260925 15:12:53.905303 13945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:53.898243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.1:53629 (local address 127.12.8.3:34991)
0925 15:12:53.898446 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:53.898455 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:53.899275 (+ 820us) server_negotiation.cc:407] Connection header received
0925 15:12:53.899517 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:53.899528 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:53.899587 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:53.899684 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:12:53.900359 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:53.901045 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:53.903015 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:53.904234 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:53.904380 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:53.904485 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:53.904890 (+ 405us) server_negotiation.cc:299] Negotiation successful
0925 15:12:53.905045 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 15:12:53.905309 13947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:53.898243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.1:56079 (local address 127.12.8.2:42187)
0925 15:12:53.898493 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:53.898499 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:53.899228 (+ 729us) server_negotiation.cc:407] Connection header received
0925 15:12:53.899518 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:53.899526 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:53.899588 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:53.899691 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:12:53.900235 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:53.901001 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:53.903110 (+ 2109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:53.904303 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:53.904427 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:53.904522 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:53.904929 (+ 407us) server_negotiation.cc:299] Negotiation successful
0925 15:12:53.905046 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260925 15:12:53.905541 13957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:53.898254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.2:42187 (local address 127.12.8.1:56079)
0925 15:12:53.899118 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:53.899134 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:12:53.899379 (+ 245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:12:53.899697 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:12:53.899701 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:12:53.899733 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:12:53.900088 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:53.900097 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:53.901124 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:53.901132 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:53.902909 (+ 1777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:53.902922 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:53.904519 (+ 1597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:53.904526 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:53.904717 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:53.904817 (+ 100us) client_negotiation.cc:769] Sending connection context
0925 15:12:53.904894 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 15:12:53.904988 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":754,"mutex_wait_us":49,"spinlock_wait_cycles":75776,"thread_start_us":474,"threads_started":1}
I20260925 15:12:53.905586 13958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:53.898254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.3:34991 (local address 127.12.8.1:53629)
0925 15:12:53.899167 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:53.899181 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 15:12:53.899368 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:12:53.899697 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:12:53.899701 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:12:53.899736 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:12:53.900205 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:53.900213 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:53.901184 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:53.901189 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:53.902873 (+ 1684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:53.902884 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:53.904537 (+ 1653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:53.904542 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:53.904712 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:53.904751 (+ 39us) client_negotiation.cc:769] Sending connection context
0925 15:12:53.904852 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 15:12:53.904986 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":790,"mutex_wait_us":143,"spinlock_wait_cycles":2432,"thread_start_us":521,"threads_started":1}
I20260925 15:12:53.906625 13744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc6e71699e744f73ba056c0359f722eb" candidate_uuid: "93a588223ed64f8abf3dc71226199d7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d1a68cc0c8c4273abdb41ff336583c6" is_pre_election: true
I20260925 15:12:53.906667 13879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc6e71699e744f73ba056c0359f722eb" candidate_uuid: "93a588223ed64f8abf3dc71226199d7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6720378ff514b54b38e689fd93699ef" is_pre_election: true
I20260925 15:12:53.907125 13744 raft_consensus.cc:2454] T dc6e71699e744f73ba056c0359f722eb P 0d1a68cc0c8c4273abdb41ff336583c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93a588223ed64f8abf3dc71226199d7c in term 0.
I20260925 15:12:53.907133 13879 raft_consensus.cc:2454] T dc6e71699e744f73ba056c0359f722eb P f6720378ff514b54b38e689fd93699ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93a588223ed64f8abf3dc71226199d7c in term 0.
I20260925 15:12:53.907790 13532 leader_election.cc:302] T dc6e71699e744f73ba056c0359f722eb P 93a588223ed64f8abf3dc71226199d7c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d1a68cc0c8c4273abdb41ff336583c6, 93a588223ed64f8abf3dc71226199d7c; no voters:
I20260925 15:12:53.908188 13955 raft_consensus.cc:2781] T dc6e71699e744f73ba056c0359f722eb P 93a588223ed64f8abf3dc71226199d7c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:12:53.908349 13955 raft_consensus.cc:482] T dc6e71699e744f73ba056c0359f722eb P 93a588223ed64f8abf3dc71226199d7c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:12:53.908449 13955 raft_consensus.cc:3037] T dc6e71699e744f73ba056c0359f722eb P 93a588223ed64f8abf3dc71226199d7c [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:12:53.910633 13955 raft_consensus.cc:504] T dc6e71699e744f73ba056c0359f722eb P 93a588223ed64f8abf3dc71226199d7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a588223ed64f8abf3dc71226199d7c" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 45655 } } peers { permanent_uuid: "f6720378ff514b54b38e689fd93699ef" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 34991 } } peers { permanent_uuid: "0d1a68cc0c8c4273abdb41ff336583c6" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 42187 } }
I20260925 15:12:53.911194 13955 leader_election.cc:288] T dc6e71699e744f73ba056c0359f722eb P 93a588223ed64f8abf3dc71226199d7c [CANDIDATE]: Term 1 election: Requested vote from peers f6720378ff514b54b38e689fd93699ef (127.12.8.3:34991), 0d1a68cc0c8c4273abdb41ff336583c6 (127.12.8.2:42187)
I20260925 15:12:53.911523 13879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc6e71699e744f73ba056c0359f722eb" candidate_uuid: "93a588223ed64f8abf3dc71226199d7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6720378ff514b54b38e689fd93699ef"
I20260925 15:12:53.911633 13744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc6e71699e744f73ba056c0359f722eb" candidate_uuid: "93a588223ed64f8abf3dc71226199d7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d1a68cc0c8c4273abdb41ff336583c6"
I20260925 15:12:53.911780 13879 raft_consensus.cc:3037] T dc6e71699e744f73ba056c0359f722eb P f6720378ff514b54b38e689fd93699ef [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:12:53.911899 13744 raft_consensus.cc:3037] T dc6e71699e744f73ba056c0359f722eb P 0d1a68cc0c8c4273abdb41ff336583c6 [term 0 FOLLOWER]: Advancing to term 1
W20260925 15:12:53.913754 13652 tablet.cc:2369] T dc6e71699e744f73ba056c0359f722eb P 93a588223ed64f8abf3dc71226199d7c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:12:53.914048 13744 raft_consensus.cc:2454] T dc6e71699e744f73ba056c0359f722eb P 0d1a68cc0c8c4273abdb41ff336583c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93a588223ed64f8abf3dc71226199d7c in term 1.
I20260925 15:12:53.914232 13879 raft_consensus.cc:2454] T dc6e71699e744f73ba056c0359f722eb P f6720378ff514b54b38e689fd93699ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93a588223ed64f8abf3dc71226199d7c in term 1.
I20260925 15:12:53.914459 13532 leader_election.cc:302] T dc6e71699e744f73ba056c0359f722eb P 93a588223ed64f8abf3dc71226199d7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d1a68cc0c8c4273abdb41ff336583c6, 93a588223ed64f8abf3dc71226199d7c; no voters:
I20260925 15:12:53.914762 13955 raft_consensus.cc:2781] T dc6e71699e744f73ba056c0359f722eb P 93a588223ed64f8abf3dc71226199d7c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:12:53.915143 13955 raft_consensus.cc:686] T dc6e71699e744f73ba056c0359f722eb P 93a588223ed64f8abf3dc71226199d7c [term 1 LEADER]: Becoming Leader. State: Replica: 93a588223ed64f8abf3dc71226199d7c, State: Running, Role: LEADER
I20260925 15:12:53.915560 13955 consensus_queue.cc:237] T dc6e71699e744f73ba056c0359f722eb P 93a588223ed64f8abf3dc71226199d7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a588223ed64f8abf3dc71226199d7c" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 45655 } } peers { permanent_uuid: "f6720378ff514b54b38e689fd93699ef" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 34991 } } peers { permanent_uuid: "0d1a68cc0c8c4273abdb41ff336583c6" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 42187 } }
I20260925 15:12:53.919649 13443 catalog_manager.cc:5495] T dc6e71699e744f73ba056c0359f722eb P 93a588223ed64f8abf3dc71226199d7c reported cstate change: term changed from 0 to 1, leader changed from <none> to 93a588223ed64f8abf3dc71226199d7c (127.12.8.1). New cstate: current_term: 1 leader_uuid: "93a588223ed64f8abf3dc71226199d7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a588223ed64f8abf3dc71226199d7c" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 45655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6720378ff514b54b38e689fd93699ef" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 34991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d1a68cc0c8c4273abdb41ff336583c6" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 42187 } health_report { overall_health: UNKNOWN } } }
W20260925 15:12:53.920683 13930 tablet.cc:2369] T dc6e71699e744f73ba056c0359f722eb P f6720378ff514b54b38e689fd93699ef: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:12:53.922551 13791 tablet.cc:2369] T dc6e71699e744f73ba056c0359f722eb P 0d1a68cc0c8c4273abdb41ff336583c6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:12:53.957938 13944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:53.952456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60436 (local address 127.12.8.1:45655)
0925 15:12:53.952669 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:53.952676 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:53.952803 (+ 127us) server_negotiation.cc:407] Connection header received
0925 15:12:53.953121 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:53.953125 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:53.953213 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:53.953352 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:12:53.953719 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:53.954491 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:53.955798 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:53.956211 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:53.956338 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:53.956841 (+ 503us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:12:53.957324 (+ 483us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:12:53.957485 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:53.957595 (+ 110us) server_negotiation.cc:299] Negotiation successful
0925 15:12:53.957671 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 15:12:53.965638 13879 raft_consensus.cc:1261] T dc6e71699e744f73ba056c0359f722eb P f6720378ff514b54b38e689fd93699ef [term 1 FOLLOWER]: Refusing update from remote peer 93a588223ed64f8abf3dc71226199d7c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:12:53.965638 13744 raft_consensus.cc:1261] T dc6e71699e744f73ba056c0359f722eb P 0d1a68cc0c8c4273abdb41ff336583c6 [term 1 FOLLOWER]: Refusing update from remote peer 93a588223ed64f8abf3dc71226199d7c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:12:53.966655 13959 consensus_queue.cc:1035] T dc6e71699e744f73ba056c0359f722eb P 93a588223ed64f8abf3dc71226199d7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d1a68cc0c8c4273abdb41ff336583c6" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:12:53.967011 13955 consensus_queue.cc:1035] T dc6e71699e744f73ba056c0359f722eb P 93a588223ed64f8abf3dc71226199d7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6720378ff514b54b38e689fd93699ef" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 34991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:12:53.980737 13962 mvcc.cc:205] Tried to move back new op lower bound from 7333270216554156032 to 7333270216361541632. Current Snapshot: MvccSnapshot[applied={T|T < 7333270216554156032}]
I20260925 15:12:54.000092 13444 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38736:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:12:54.000347 13444 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38736:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:12:54.003959 13444 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6676713ee26a4eb188da557ac038061e: Sending DeleteTablet for 3 replicas of tablet dc6e71699e744f73ba056c0359f722eb
I20260925 15:12:54.005306 13586 tablet_service.cc:1514] Processing DeleteTablet for tablet dc6e71699e744f73ba056c0359f722eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:12:53 UTC) from {username='kudu', principal='kudu/127.12.8.62@KRBTEST.COM'} at 127.0.0.1:60428
I20260925 15:12:54.005261 13859 tablet_service.cc:1514] Processing DeleteTablet for tablet dc6e71699e744f73ba056c0359f722eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:12:53 UTC) from {username='kudu', principal='kudu/127.12.8.62@KRBTEST.COM'} at 127.0.0.1:41780
I20260925 15:12:54.005981 13720 tablet_service.cc:1514] Processing DeleteTablet for tablet dc6e71699e744f73ba056c0359f722eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:12:53 UTC) from {username='kudu', principal='kudu/127.12.8.62@KRBTEST.COM'} at 127.0.0.1:41926
I20260925 15:12:54.006963 13973 tablet_replica.cc:330] T dc6e71699e744f73ba056c0359f722eb P 0d1a68cc0c8c4273abdb41ff336583c6: stopping tablet replica
I20260925 15:12:54.007421 13973 raft_consensus.cc:2229] T dc6e71699e744f73ba056c0359f722eb P 0d1a68cc0c8c4273abdb41ff336583c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:12:54.007598 13444 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38736:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 15:12:54.008000 13973 raft_consensus.cc:2258] T dc6e71699e744f73ba056c0359f722eb P 0d1a68cc0c8c4273abdb41ff336583c6 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260925 15:12:54.008414 13444 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:12:54.010080 13973 ts_tablet_manager.cc:1905] T dc6e71699e744f73ba056c0359f722eb P 0d1a68cc0c8c4273abdb41ff336583c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:12:54.017568 13973 ts_tablet_manager.cc:1918] T dc6e71699e744f73ba056c0359f722eb P 0d1a68cc0c8c4273abdb41ff336583c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:12:54.017835 13973 log.cc:1196] T dc6e71699e744f73ba056c0359f722eb P 0d1a68cc0c8c4273abdb41ff336583c6: Deleting WAL directory at /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/minicluster-data/ts-1/wal/wals/dc6e71699e744f73ba056c0359f722eb
I20260925 15:12:54.018474 13973 ts_tablet_manager.cc:1939] T dc6e71699e744f73ba056c0359f722eb P 0d1a68cc0c8c4273abdb41ff336583c6: Deleting consensus metadata
I20260925 15:12:54.020383 13429 catalog_manager.cc:4841] TS 0d1a68cc0c8c4273abdb41ff336583c6 (127.12.8.2:42187): tablet dc6e71699e744f73ba056c0359f722eb (table test-table [id=598259c1c82540ea8c690ca4ed9f50d7]) successfully deleted
I20260925 15:12:54.022559 13585 tablet_service.cc:1467] Processing CreateTablet for tablet fea290f10c34443f8c9727d62bb12375 (DEFAULT_TABLE table=test-table [id=6e30f4f916f94be7bf9b15a9a2721067]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:12:54.022921 13720 tablet_service.cc:1467] Processing CreateTablet for tablet fea290f10c34443f8c9727d62bb12375 (DEFAULT_TABLE table=test-table [id=6e30f4f916f94be7bf9b15a9a2721067]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:12:54.023015 13858 tablet_service.cc:1467] Processing CreateTablet for tablet fea290f10c34443f8c9727d62bb12375 (DEFAULT_TABLE table=test-table [id=6e30f4f916f94be7bf9b15a9a2721067]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:12:54.023185 13585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fea290f10c34443f8c9727d62bb12375. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:12:54.023320 13720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fea290f10c34443f8c9727d62bb12375. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:12:54.023548 13858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fea290f10c34443f8c9727d62bb12375. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:12:54.029917 13950 tablet_bootstrap.cc:492] T fea290f10c34443f8c9727d62bb12375 P 0d1a68cc0c8c4273abdb41ff336583c6: Bootstrap starting.
I20260925 15:12:54.031548 13950 tablet_bootstrap.cc:654] T fea290f10c34443f8c9727d62bb12375 P 0d1a68cc0c8c4273abdb41ff336583c6: Neither blocks nor log segments found. Creating new log.
I20260925 15:12:54.031826 13949 tablet_bootstrap.cc:492] T fea290f10c34443f8c9727d62bb12375 P f6720378ff514b54b38e689fd93699ef: Bootstrap starting.
I20260925 15:12:54.033900 13949 tablet_bootstrap.cc:654] T fea290f10c34443f8c9727d62bb12375 P f6720378ff514b54b38e689fd93699ef: Neither blocks nor log segments found. Creating new log.
I20260925 15:12:54.035251 13948 tablet_bootstrap.cc:492] T fea290f10c34443f8c9727d62bb12375 P 93a588223ed64f8abf3dc71226199d7c: Bootstrap starting.
I20260925 15:12:54.035331 13950 tablet_bootstrap.cc:492] T fea290f10c34443f8c9727d62bb12375 P 0d1a68cc0c8c4273abdb41ff336583c6: No bootstrap required, opened a new log
I20260925 15:12:54.035457 13950 ts_tablet_manager.cc:1397] T fea290f10c34443f8c9727d62bb12375 P 0d1a68cc0c8c4273abdb41ff336583c6: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 15:12:54.036223 13972 tablet_replica.cc:330] T dc6e71699e744f73ba056c0359f722eb P 93a588223ed64f8abf3dc71226199d7c: stopping tablet replica
I20260925 15:12:54.036320 13950 raft_consensus.cc:348] T fea290f10c34443f8c9727d62bb12375 P 0d1a68cc0c8c4273abdb41ff336583c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a588223ed64f8abf3dc71226199d7c" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 45655 } } peers { permanent_uuid: "0d1a68cc0c8c4273abdb41ff336583c6" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 42187 } } peers { permanent_uuid: "f6720378ff514b54b38e689fd93699ef" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 34991 } }
I20260925 15:12:54.036509 13950 raft_consensus.cc:374] T fea290f10c34443f8c9727d62bb12375 P 0d1a68cc0c8c4273abdb41ff336583c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:12:54.036586 13950 raft_consensus.cc:729] T fea290f10c34443f8c9727d62bb12375 P 0d1a68cc0c8c4273abdb41ff336583c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d1a68cc0c8c4273abdb41ff336583c6, State: Initialized, Role: FOLLOWER
I20260925 15:12:54.036715 13972 raft_consensus.cc:2229] T dc6e71699e744f73ba056c0359f722eb P 93a588223ed64f8abf3dc71226199d7c [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:12:54.036931 13948 tablet_bootstrap.cc:654] T fea290f10c34443f8c9727d62bb12375 P 93a588223ed64f8abf3dc71226199d7c: Neither blocks nor log segments found. Creating new log.
I20260925 15:12:54.036837 13950 consensus_queue.cc:260] T fea290f10c34443f8c9727d62bb12375 P 0d1a68cc0c8c4273abdb41ff336583c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a588223ed64f8abf3dc71226199d7c" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 45655 } } peers { permanent_uuid: "0d1a68cc0c8c4273abdb41ff336583c6" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 42187 } } peers { permanent_uuid: "f6720378ff514b54b38e689fd93699ef" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 34991 } }
I20260925 15:12:54.037355 13950 ts_tablet_manager.cc:1428] T fea290f10c34443f8c9727d62bb12375 P 0d1a68cc0c8c4273abdb41ff336583c6: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260925 15:12:54.037364 13972 raft_consensus.cc:2258] T dc6e71699e744f73ba056c0359f722eb P 93a588223ed64f8abf3dc71226199d7c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:12:54.037652 13971 tablet_replica.cc:330] T dc6e71699e744f73ba056c0359f722eb P f6720378ff514b54b38e689fd93699ef: stopping tablet replica
I20260925 15:12:54.038372 13971 raft_consensus.cc:2229] T dc6e71699e744f73ba056c0359f722eb P f6720378ff514b54b38e689fd93699ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:12:54.038898 13971 raft_consensus.cc:2258] T dc6e71699e744f73ba056c0359f722eb P f6720378ff514b54b38e689fd93699ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:12:54.041060 13971 ts_tablet_manager.cc:1905] T dc6e71699e744f73ba056c0359f722eb P f6720378ff514b54b38e689fd93699ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:12:54.042290 13949 tablet_bootstrap.cc:492] T fea290f10c34443f8c9727d62bb12375 P f6720378ff514b54b38e689fd93699ef: No bootstrap required, opened a new log
I20260925 15:12:54.042443 13949 ts_tablet_manager.cc:1397] T fea290f10c34443f8c9727d62bb12375 P f6720378ff514b54b38e689fd93699ef: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20260925 15:12:54.042466 13972 ts_tablet_manager.cc:1905] T dc6e71699e744f73ba056c0359f722eb P 93a588223ed64f8abf3dc71226199d7c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:12:54.043560 13949 raft_consensus.cc:348] T fea290f10c34443f8c9727d62bb12375 P f6720378ff514b54b38e689fd93699ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a588223ed64f8abf3dc71226199d7c" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 45655 } } peers { permanent_uuid: "0d1a68cc0c8c4273abdb41ff336583c6" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 42187 } } peers { permanent_uuid: "f6720378ff514b54b38e689fd93699ef" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 34991 } }
I20260925 15:12:54.043875 13949 raft_consensus.cc:374] T fea290f10c34443f8c9727d62bb12375 P f6720378ff514b54b38e689fd93699ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:12:54.043982 13949 raft_consensus.cc:729] T fea290f10c34443f8c9727d62bb12375 P f6720378ff514b54b38e689fd93699ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6720378ff514b54b38e689fd93699ef, State: Initialized, Role: FOLLOWER
I20260925 15:12:54.045341 13949 consensus_queue.cc:260] T fea290f10c34443f8c9727d62bb12375 P f6720378ff514b54b38e689fd93699ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a588223ed64f8abf3dc71226199d7c" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 45655 } } peers { permanent_uuid: "0d1a68cc0c8c4273abdb41ff336583c6" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 42187 } } peers { permanent_uuid: "f6720378ff514b54b38e689fd93699ef" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 34991 } }
I20260925 15:12:54.045979 13949 ts_tablet_manager.cc:1428] T fea290f10c34443f8c9727d62bb12375 P f6720378ff514b54b38e689fd93699ef: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260925 15:12:54.048527 13971 ts_tablet_manager.cc:1918] T dc6e71699e744f73ba056c0359f722eb P f6720378ff514b54b38e689fd93699ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:12:54.048723 13971 log.cc:1196] T dc6e71699e744f73ba056c0359f722eb P f6720378ff514b54b38e689fd93699ef: Deleting WAL directory at /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/minicluster-data/ts-2/wal/wals/dc6e71699e744f73ba056c0359f722eb
I20260925 15:12:54.049315 13971 ts_tablet_manager.cc:1939] T dc6e71699e744f73ba056c0359f722eb P f6720378ff514b54b38e689fd93699ef: Deleting consensus metadata
I20260925 15:12:54.051679 13431 catalog_manager.cc:4841] TS f6720378ff514b54b38e689fd93699ef (127.12.8.3:34991): tablet dc6e71699e744f73ba056c0359f722eb (table test-table [id=598259c1c82540ea8c690ca4ed9f50d7]) successfully deleted
I20260925 15:12:54.053671 13972 ts_tablet_manager.cc:1918] T dc6e71699e744f73ba056c0359f722eb P 93a588223ed64f8abf3dc71226199d7c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:12:54.053838 13972 log.cc:1196] T dc6e71699e744f73ba056c0359f722eb P 93a588223ed64f8abf3dc71226199d7c: Deleting WAL directory at /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/minicluster-data/ts-0/wal/wals/dc6e71699e744f73ba056c0359f722eb
I20260925 15:12:54.054392 13972 ts_tablet_manager.cc:1939] T dc6e71699e744f73ba056c0359f722eb P 93a588223ed64f8abf3dc71226199d7c: Deleting consensus metadata
I20260925 15:12:54.056059 13428 catalog_manager.cc:4841] TS 93a588223ed64f8abf3dc71226199d7c (127.12.8.1:45655): tablet dc6e71699e744f73ba056c0359f722eb (table test-table [id=598259c1c82540ea8c690ca4ed9f50d7]) successfully deleted
I20260925 15:12:54.057489 13948 tablet_bootstrap.cc:492] T fea290f10c34443f8c9727d62bb12375 P 93a588223ed64f8abf3dc71226199d7c: No bootstrap required, opened a new log
I20260925 15:12:54.057636 13948 ts_tablet_manager.cc:1397] T fea290f10c34443f8c9727d62bb12375 P 93a588223ed64f8abf3dc71226199d7c: Time spent bootstrapping tablet: real 0.022s user 0.006s sys 0.006s
I20260925 15:12:54.058499 13948 raft_consensus.cc:348] T fea290f10c34443f8c9727d62bb12375 P 93a588223ed64f8abf3dc71226199d7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a588223ed64f8abf3dc71226199d7c" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 45655 } } peers { permanent_uuid: "0d1a68cc0c8c4273abdb41ff336583c6" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 42187 } } peers { permanent_uuid: "f6720378ff514b54b38e689fd93699ef" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 34991 } }
I20260925 15:12:54.058751 13948 raft_consensus.cc:374] T fea290f10c34443f8c9727d62bb12375 P 93a588223ed64f8abf3dc71226199d7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:12:54.058861 13948 raft_consensus.cc:729] T fea290f10c34443f8c9727d62bb12375 P 93a588223ed64f8abf3dc71226199d7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93a588223ed64f8abf3dc71226199d7c, State: Initialized, Role: FOLLOWER
I20260925 15:12:54.059140 13948 consensus_queue.cc:260] T fea290f10c34443f8c9727d62bb12375 P 93a588223ed64f8abf3dc71226199d7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a588223ed64f8abf3dc71226199d7c" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 45655 } } peers { permanent_uuid: "0d1a68cc0c8c4273abdb41ff336583c6" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 42187 } } peers { permanent_uuid: "f6720378ff514b54b38e689fd93699ef" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 34991 } }
I20260925 15:12:54.059824 13948 ts_tablet_manager.cc:1428] T fea290f10c34443f8c9727d62bb12375 P 93a588223ed64f8abf3dc71226199d7c: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 15:12:54.188966 13956 raft_consensus.cc:482] T fea290f10c34443f8c9727d62bb12375 P 0d1a68cc0c8c4273abdb41ff336583c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:12:54.189195 13956 raft_consensus.cc:504] T fea290f10c34443f8c9727d62bb12375 P 0d1a68cc0c8c4273abdb41ff336583c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a588223ed64f8abf3dc71226199d7c" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 45655 } } peers { permanent_uuid: "0d1a68cc0c8c4273abdb41ff336583c6" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 42187 } } peers { permanent_uuid: "f6720378ff514b54b38e689fd93699ef" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 34991 } }
I20260925 15:12:54.190837 13956 leader_election.cc:288] T fea290f10c34443f8c9727d62bb12375 P 0d1a68cc0c8c4273abdb41ff336583c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93a588223ed64f8abf3dc71226199d7c (127.12.8.1:45655), f6720378ff514b54b38e689fd93699ef (127.12.8.3:34991)
I20260925 15:12:54.199293 13945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:54.191335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.2:47489 (local address 127.12.8.3:34991)
0925 15:12:54.191575 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:54.191584 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:54.192851 (+ 1267us) server_negotiation.cc:407] Connection header received
0925 15:12:54.193023 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:54.193028 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:54.193098 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:54.193182 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:12:54.193957 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:54.195076 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:54.197482 (+ 2406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:54.198311 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:54.198386 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:54.198466 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:54.198917 (+ 451us) server_negotiation.cc:299] Negotiation successful
0925 15:12:54.199039 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260925 15:12:54.199429 13978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:54.191161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.3:34991 (local address 127.12.8.2:47489)
0925 15:12:54.192738 (+ 1577us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:54.192760 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 15:12:54.192889 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:12:54.193214 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:12:54.193219 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:12:54.193268 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:12:54.193763 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:54.193780 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:54.195212 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:54.195220 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:54.197308 (+ 2088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:54.197321 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:54.198614 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:54.198618 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:54.198757 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:54.198779 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 15:12:54.198886 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 15:12:54.199007 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1445,"mutex_wait_us":21,"spinlock_wait_cycles":896,"thread_start_us":953,"threads_started":1}
I20260925 15:12:54.200119 13879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fea290f10c34443f8c9727d62bb12375" candidate_uuid: "0d1a68cc0c8c4273abdb41ff336583c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6720378ff514b54b38e689fd93699ef" is_pre_election: true
I20260925 15:12:54.200378 13879 raft_consensus.cc:2454] T fea290f10c34443f8c9727d62bb12375 P f6720378ff514b54b38e689fd93699ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d1a68cc0c8c4273abdb41ff336583c6 in term 0.
I20260925 15:12:54.200407 13944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:54.191752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.2:48617 (local address 127.12.8.1:45655)
0925 15:12:54.191950 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:54.191958 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:54.192565 (+ 607us) server_negotiation.cc:407] Connection header received
0925 15:12:54.192823 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:54.192831 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:54.192922 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:54.193031 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:12:54.193720 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:54.194772 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:54.197449 (+ 2677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:54.198741 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:54.198892 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:54.198987 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:54.200011 (+ 1024us) server_negotiation.cc:299] Negotiation successful
0925 15:12:54.200132 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 15:12:54.200474 13977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:54.191132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.1:45655 (local address 127.12.8.2:48617)
0925 15:12:54.192461 (+ 1329us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:54.192480 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 15:12:54.192676 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:12:54.193089 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:12:54.193094 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:12:54.193125 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:12:54.193538 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:54.193551 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:54.194896 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:54.194904 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:54.197261 (+ 2357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:54.197279 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:54.199670 (+ 2391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:54.199679 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:54.199847 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:54.199870 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 15:12:54.199977 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 15:12:54.200092 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1156,"mutex_wait_us":24,"spinlock_wait_cycles":53760,"thread_start_us":1275,"threads_started":1}
I20260925 15:12:54.201157 13674 leader_election.cc:302] T fea290f10c34443f8c9727d62bb12375 P 0d1a68cc0c8c4273abdb41ff336583c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d1a68cc0c8c4273abdb41ff336583c6, f6720378ff514b54b38e689fd93699ef; no voters:
I20260925 15:12:54.201572 13956 raft_consensus.cc:2781] T fea290f10c34443f8c9727d62bb12375 P 0d1a68cc0c8c4273abdb41ff336583c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:12:54.201676 13956 raft_consensus.cc:482] T fea290f10c34443f8c9727d62bb12375 P 0d1a68cc0c8c4273abdb41ff336583c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:12:54.201787 13956 raft_consensus.cc:3037] T fea290f10c34443f8c9727d62bb12375 P 0d1a68cc0c8c4273abdb41ff336583c6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:12:54.201701 13606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fea290f10c34443f8c9727d62bb12375" candidate_uuid: "0d1a68cc0c8c4273abdb41ff336583c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93a588223ed64f8abf3dc71226199d7c" is_pre_election: true
I20260925 15:12:54.202139 13606 raft_consensus.cc:2454] T fea290f10c34443f8c9727d62bb12375 P 93a588223ed64f8abf3dc71226199d7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d1a68cc0c8c4273abdb41ff336583c6 in term 0.
I20260925 15:12:54.204258 13956 raft_consensus.cc:504] T fea290f10c34443f8c9727d62bb12375 P 0d1a68cc0c8c4273abdb41ff336583c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a588223ed64f8abf3dc71226199d7c" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 45655 } } peers { permanent_uuid: "0d1a68cc0c8c4273abdb41ff336583c6" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 42187 } } peers { permanent_uuid: "f6720378ff514b54b38e689fd93699ef" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 34991 } }
I20260925 15:12:54.204943 13956 leader_election.cc:288] T fea290f10c34443f8c9727d62bb12375 P 0d1a68cc0c8c4273abdb41ff336583c6 [CANDIDATE]: Term 1 election: Requested vote from peers 93a588223ed64f8abf3dc71226199d7c (127.12.8.1:45655), f6720378ff514b54b38e689fd93699ef (127.12.8.3:34991)
I20260925 15:12:54.205348 13606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fea290f10c34443f8c9727d62bb12375" candidate_uuid: "0d1a68cc0c8c4273abdb41ff336583c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93a588223ed64f8abf3dc71226199d7c"
I20260925 15:12:54.205444 13879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fea290f10c34443f8c9727d62bb12375" candidate_uuid: "0d1a68cc0c8c4273abdb41ff336583c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6720378ff514b54b38e689fd93699ef"
I20260925 15:12:54.205546 13606 raft_consensus.cc:3037] T fea290f10c34443f8c9727d62bb12375 P 93a588223ed64f8abf3dc71226199d7c [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:12:54.205663 13879 raft_consensus.cc:3037] T fea290f10c34443f8c9727d62bb12375 P f6720378ff514b54b38e689fd93699ef [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:12:54.207825 13606 raft_consensus.cc:2454] T fea290f10c34443f8c9727d62bb12375 P 93a588223ed64f8abf3dc71226199d7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d1a68cc0c8c4273abdb41ff336583c6 in term 1.
I20260925 15:12:54.208220 13671 leader_election.cc:302] T fea290f10c34443f8c9727d62bb12375 P 0d1a68cc0c8c4273abdb41ff336583c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d1a68cc0c8c4273abdb41ff336583c6, 93a588223ed64f8abf3dc71226199d7c; no voters:
I20260925 15:12:54.208484 13956 raft_consensus.cc:2781] T fea290f10c34443f8c9727d62bb12375 P 0d1a68cc0c8c4273abdb41ff336583c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:12:54.208519 13879 raft_consensus.cc:2454] T fea290f10c34443f8c9727d62bb12375 P f6720378ff514b54b38e689fd93699ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d1a68cc0c8c4273abdb41ff336583c6 in term 1.
I20260925 15:12:54.219580 13956 raft_consensus.cc:686] T fea290f10c34443f8c9727d62bb12375 P 0d1a68cc0c8c4273abdb41ff336583c6 [term 1 LEADER]: Becoming Leader. State: Replica: 0d1a68cc0c8c4273abdb41ff336583c6, State: Running, Role: LEADER
I20260925 15:12:54.219974 13956 consensus_queue.cc:237] T fea290f10c34443f8c9727d62bb12375 P 0d1a68cc0c8c4273abdb41ff336583c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a588223ed64f8abf3dc71226199d7c" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 45655 } } peers { permanent_uuid: "0d1a68cc0c8c4273abdb41ff336583c6" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 42187 } } peers { permanent_uuid: "f6720378ff514b54b38e689fd93699ef" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 34991 } }
I20260925 15:12:54.223134 13443 catalog_manager.cc:5495] T fea290f10c34443f8c9727d62bb12375 P 0d1a68cc0c8c4273abdb41ff336583c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d1a68cc0c8c4273abdb41ff336583c6 (127.12.8.2). New cstate: current_term: 1 leader_uuid: "0d1a68cc0c8c4273abdb41ff336583c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a588223ed64f8abf3dc71226199d7c" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 45655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d1a68cc0c8c4273abdb41ff336583c6" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 42187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6720378ff514b54b38e689fd93699ef" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 34991 } health_report { overall_health: UNKNOWN } } }
Sep 25 15:12:54 dist-test-slave-p468 krb5kdc[13399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349172, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.62@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
I20260925 15:12:54.263803 13986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:54.248448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38740 (local address 127.12.8.62:39285)
0925 15:12:54.249135 (+ 687us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:54.249146 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:54.249434 (+ 288us) server_negotiation.cc:407] Connection header received
0925 15:12:54.249610 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:54.249614 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:54.249670 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:54.249749 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:54.251245 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:54.252012 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:54.253792 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:54.254175 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:54.254326 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:54.258321 (+ 3995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:54.258326 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:54.258329 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:12:54.258357 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:54.261156 (+ 2799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:54.261988 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:54.261992 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:54.261994 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:54.262104 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:54.262525 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:54.262529 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:54.262531 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:54.262771 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:54.262887 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:54.263368 (+ 481us) server_negotiation.cc:299] Negotiation successful
0925 15:12:54.263563 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":515,"thread_start_us":308,"threads_started":1}
I20260925 15:12:54.264178 13987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:54.248436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:39285 (local address 127.0.0.1:38740)
0925 15:12:54.249319 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:54.249339 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 15:12:54.249487 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:12:54.249831 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:12:54.249836 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:12:54.250747 (+ 911us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:12:54.251063 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:54.251077 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:54.252144 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:54.252149 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:54.253661 (+ 1512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:54.253678 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:54.254407 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:54.254414 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:54.254591 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:54.255266 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:12:54.255291 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:12:54.258078 (+ 2787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:12:54.261297 (+ 3219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:12:54.261308 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:12:54.261316 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:12:54.261872 (+ 556us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:12:54.262234 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:12:54.262239 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:12:54.262243 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:12:54.262414 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:12:54.262893 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:12:54.262902 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:12:54.263104 (+ 202us) client_negotiation.cc:769] Sending connection context
0925 15:12:54.263301 (+ 197us) client_negotiation.cc:240] Negotiation successful
0925 15:12:54.263728 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":715,"mutex_wait_us":124,"thread_start_us":566,"threads_started":1}
I20260925 15:12:54.269148 13443 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.12.8.62@KRBTEST.COM'} at 127.0.0.1:38740:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 15:12:54.270117 13443 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:12:54.276816 13586 tablet_service.cc:1467] Processing CreateTablet for tablet 864b70889e434a37946e2b4040a5f1fe (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e49b113af06c46b8ae450a73e3ac6392]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 15:12:54.277079 13720 tablet_service.cc:1467] Processing CreateTablet for tablet 864b70889e434a37946e2b4040a5f1fe (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e49b113af06c46b8ae450a73e3ac6392]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 15:12:54.277438 13586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 864b70889e434a37946e2b4040a5f1fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:12:54.277336 13859 tablet_service.cc:1467] Processing CreateTablet for tablet 864b70889e434a37946e2b4040a5f1fe (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e49b113af06c46b8ae450a73e3ac6392]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 15:12:54.277563 13720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 864b70889e434a37946e2b4040a5f1fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:12:54.277920 13859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 864b70889e434a37946e2b4040a5f1fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:12:54.284983 13948 tablet_bootstrap.cc:492] T 864b70889e434a37946e2b4040a5f1fe P 93a588223ed64f8abf3dc71226199d7c: Bootstrap starting.
I20260925 15:12:54.285061 13949 tablet_bootstrap.cc:492] T 864b70889e434a37946e2b4040a5f1fe P f6720378ff514b54b38e689fd93699ef: Bootstrap starting.
I20260925 15:12:54.286715 13948 tablet_bootstrap.cc:654] T 864b70889e434a37946e2b4040a5f1fe P 93a588223ed64f8abf3dc71226199d7c: Neither blocks nor log segments found. Creating new log.
I20260925 15:12:54.286713 13949 tablet_bootstrap.cc:654] T 864b70889e434a37946e2b4040a5f1fe P f6720378ff514b54b38e689fd93699ef: Neither blocks nor log segments found. Creating new log.
I20260925 15:12:54.287096 13950 tablet_bootstrap.cc:492] T 864b70889e434a37946e2b4040a5f1fe P 0d1a68cc0c8c4273abdb41ff336583c6: Bootstrap starting.
I20260925 15:12:54.288728 13950 tablet_bootstrap.cc:654] T 864b70889e434a37946e2b4040a5f1fe P 0d1a68cc0c8c4273abdb41ff336583c6: Neither blocks nor log segments found. Creating new log.
I20260925 15:12:54.298103 13949 tablet_bootstrap.cc:492] T 864b70889e434a37946e2b4040a5f1fe P f6720378ff514b54b38e689fd93699ef: No bootstrap required, opened a new log
I20260925 15:12:54.298902 13948 tablet_bootstrap.cc:492] T 864b70889e434a37946e2b4040a5f1fe P 93a588223ed64f8abf3dc71226199d7c: No bootstrap required, opened a new log
I20260925 15:12:54.299084 13948 ts_tablet_manager.cc:1397] T 864b70889e434a37946e2b4040a5f1fe P 93a588223ed64f8abf3dc71226199d7c: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.003s
I20260925 15:12:54.299180 13950 tablet_bootstrap.cc:492] T 864b70889e434a37946e2b4040a5f1fe P 0d1a68cc0c8c4273abdb41ff336583c6: No bootstrap required, opened a new log
I20260925 15:12:54.299335 13950 ts_tablet_manager.cc:1397] T 864b70889e434a37946e2b4040a5f1fe P 0d1a68cc0c8c4273abdb41ff336583c6: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260925 15:12:54.299909 13948 raft_consensus.cc:348] T 864b70889e434a37946e2b4040a5f1fe P 93a588223ed64f8abf3dc71226199d7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a588223ed64f8abf3dc71226199d7c" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 45655 } } peers { permanent_uuid: "0d1a68cc0c8c4273abdb41ff336583c6" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 42187 } } peers { permanent_uuid: "f6720378ff514b54b38e689fd93699ef" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 34991 } }
I20260925 15:12:54.299993 13950 raft_consensus.cc:348] T 864b70889e434a37946e2b4040a5f1fe P 0d1a68cc0c8c4273abdb41ff336583c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a588223ed64f8abf3dc71226199d7c" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 45655 } } peers { permanent_uuid: "0d1a68cc0c8c4273abdb41ff336583c6" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 42187 } } peers { permanent_uuid: "f6720378ff514b54b38e689fd93699ef" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 34991 } }
I20260925 15:12:54.300166 13948 raft_consensus.cc:374] T 864b70889e434a37946e2b4040a5f1fe P 93a588223ed64f8abf3dc71226199d7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:12:54.300207 13950 raft_consensus.cc:374] T 864b70889e434a37946e2b4040a5f1fe P 0d1a68cc0c8c4273abdb41ff336583c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:12:54.300256 13948 raft_consensus.cc:729] T 864b70889e434a37946e2b4040a5f1fe P 93a588223ed64f8abf3dc71226199d7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93a588223ed64f8abf3dc71226199d7c, State: Initialized, Role: FOLLOWER
I20260925 15:12:54.300292 13950 raft_consensus.cc:729] T 864b70889e434a37946e2b4040a5f1fe P 0d1a68cc0c8c4273abdb41ff336583c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d1a68cc0c8c4273abdb41ff336583c6, State: Initialized, Role: FOLLOWER
I20260925 15:12:54.300561 13949 ts_tablet_manager.cc:1397] T 864b70889e434a37946e2b4040a5f1fe P f6720378ff514b54b38e689fd93699ef: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.003s
I20260925 15:12:54.300560 13950 consensus_queue.cc:260] T 864b70889e434a37946e2b4040a5f1fe P 0d1a68cc0c8c4273abdb41ff336583c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a588223ed64f8abf3dc71226199d7c" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 45655 } } peers { permanent_uuid: "0d1a68cc0c8c4273abdb41ff336583c6" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 42187 } } peers { permanent_uuid: "f6720378ff514b54b38e689fd93699ef" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 34991 } }
I20260925 15:12:54.300999 13956 tablet_replica.cc:439] T 864b70889e434a37946e2b4040a5f1fe P 0d1a68cc0c8c4273abdb41ff336583c6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a588223ed64f8abf3dc71226199d7c" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 45655 } } peers { permanent_uuid: "0d1a68cc0c8c4273abdb41ff336583c6" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 42187 } } peers { permanent_uuid: "f6720378ff514b54b38e689fd93699ef" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 34991 } } }
I20260925 15:12:54.301272 13956 tablet_replica.cc:442] T 864b70889e434a37946e2b4040a5f1fe P 0d1a68cc0c8c4273abdb41ff336583c6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:12:54.301545 13948 consensus_queue.cc:260] T 864b70889e434a37946e2b4040a5f1fe P 93a588223ed64f8abf3dc71226199d7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a588223ed64f8abf3dc71226199d7c" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 45655 } } peers { permanent_uuid: "0d1a68cc0c8c4273abdb41ff336583c6" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 42187 } } peers { permanent_uuid: "f6720378ff514b54b38e689fd93699ef" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 34991 } }
I20260925 15:12:54.301741 13950 ts_tablet_manager.cc:1428] T 864b70889e434a37946e2b4040a5f1fe P 0d1a68cc0c8c4273abdb41ff336583c6: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260925 15:12:54.302016 13955 tablet_replica.cc:439] T 864b70889e434a37946e2b4040a5f1fe P 93a588223ed64f8abf3dc71226199d7c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a588223ed64f8abf3dc71226199d7c" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 45655 } } peers { permanent_uuid: "0d1a68cc0c8c4273abdb41ff336583c6" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 42187 } } peers { permanent_uuid: "f6720378ff514b54b38e689fd93699ef" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 34991 } } }
I20260925 15:12:54.302250 13955 tablet_replica.cc:442] T 864b70889e434a37946e2b4040a5f1fe P 93a588223ed64f8abf3dc71226199d7c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:12:54.302703 13948 ts_tablet_manager.cc:1428] T 864b70889e434a37946e2b4040a5f1fe P 93a588223ed64f8abf3dc71226199d7c: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260925 15:12:54.303257 13949 raft_consensus.cc:348] T 864b70889e434a37946e2b4040a5f1fe P f6720378ff514b54b38e689fd93699ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a588223ed64f8abf3dc71226199d7c" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 45655 } } peers { permanent_uuid: "0d1a68cc0c8c4273abdb41ff336583c6" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 42187 } } peers { permanent_uuid: "f6720378ff514b54b38e689fd93699ef" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 34991 } }
I20260925 15:12:54.303478 13949 raft_consensus.cc:374] T 864b70889e434a37946e2b4040a5f1fe P f6720378ff514b54b38e689fd93699ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:12:54.303546 13949 raft_consensus.cc:729] T 864b70889e434a37946e2b4040a5f1fe P f6720378ff514b54b38e689fd93699ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6720378ff514b54b38e689fd93699ef, State: Initialized, Role: FOLLOWER
I20260925 15:12:54.303754 13949 consensus_queue.cc:260] T 864b70889e434a37946e2b4040a5f1fe P f6720378ff514b54b38e689fd93699ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a588223ed64f8abf3dc71226199d7c" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 45655 } } peers { permanent_uuid: "0d1a68cc0c8c4273abdb41ff336583c6" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 42187 } } peers { permanent_uuid: "f6720378ff514b54b38e689fd93699ef" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 34991 } }
I20260925 15:12:54.304366 13949 ts_tablet_manager.cc:1428] T 864b70889e434a37946e2b4040a5f1fe P f6720378ff514b54b38e689fd93699ef: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20260925 15:12:54.304330 13954 tablet_replica.cc:439] T 864b70889e434a37946e2b4040a5f1fe P f6720378ff514b54b38e689fd93699ef: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a588223ed64f8abf3dc71226199d7c" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 45655 } } peers { permanent_uuid: "0d1a68cc0c8c4273abdb41ff336583c6" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 42187 } } peers { permanent_uuid: "f6720378ff514b54b38e689fd93699ef" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 34991 } } }
I20260925 15:12:54.304682 13954 tablet_replica.cc:442] T 864b70889e434a37946e2b4040a5f1fe P f6720378ff514b54b38e689fd93699ef: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:12:54.516312 13955 raft_consensus.cc:482] T 864b70889e434a37946e2b4040a5f1fe P 93a588223ed64f8abf3dc71226199d7c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:12:54.516532 13955 raft_consensus.cc:504] T 864b70889e434a37946e2b4040a5f1fe P 93a588223ed64f8abf3dc71226199d7c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a588223ed64f8abf3dc71226199d7c" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 45655 } } peers { permanent_uuid: "0d1a68cc0c8c4273abdb41ff336583c6" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 42187 } } peers { permanent_uuid: "f6720378ff514b54b38e689fd93699ef" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 34991 } }
I20260925 15:12:54.517175 13955 leader_election.cc:288] T 864b70889e434a37946e2b4040a5f1fe P 93a588223ed64f8abf3dc71226199d7c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d1a68cc0c8c4273abdb41ff336583c6 (127.12.8.2:42187), f6720378ff514b54b38e689fd93699ef (127.12.8.3:34991)
I20260925 15:12:54.517747 13879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "864b70889e434a37946e2b4040a5f1fe" candidate_uuid: "93a588223ed64f8abf3dc71226199d7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6720378ff514b54b38e689fd93699ef" is_pre_election: true
I20260925 15:12:54.517764 13744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "864b70889e434a37946e2b4040a5f1fe" candidate_uuid: "93a588223ed64f8abf3dc71226199d7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d1a68cc0c8c4273abdb41ff336583c6" is_pre_election: true
I20260925 15:12:54.518006 13879 raft_consensus.cc:2454] T 864b70889e434a37946e2b4040a5f1fe P f6720378ff514b54b38e689fd93699ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93a588223ed64f8abf3dc71226199d7c in term 0.
I20260925 15:12:54.518002 13744 raft_consensus.cc:2454] T 864b70889e434a37946e2b4040a5f1fe P 0d1a68cc0c8c4273abdb41ff336583c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93a588223ed64f8abf3dc71226199d7c in term 0.
I20260925 15:12:54.518499 13534 leader_election.cc:302] T 864b70889e434a37946e2b4040a5f1fe P 93a588223ed64f8abf3dc71226199d7c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0d1a68cc0c8c4273abdb41ff336583c6, 93a588223ed64f8abf3dc71226199d7c, f6720378ff514b54b38e689fd93699ef; no voters:
I20260925 15:12:54.518759 13955 raft_consensus.cc:2781] T 864b70889e434a37946e2b4040a5f1fe P 93a588223ed64f8abf3dc71226199d7c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:12:54.518882 13955 raft_consensus.cc:482] T 864b70889e434a37946e2b4040a5f1fe P 93a588223ed64f8abf3dc71226199d7c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:12:54.518999 13955 raft_consensus.cc:3037] T 864b70889e434a37946e2b4040a5f1fe P 93a588223ed64f8abf3dc71226199d7c [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:12:54.520877 13955 raft_consensus.cc:504] T 864b70889e434a37946e2b4040a5f1fe P 93a588223ed64f8abf3dc71226199d7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a588223ed64f8abf3dc71226199d7c" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 45655 } } peers { permanent_uuid: "0d1a68cc0c8c4273abdb41ff336583c6" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 42187 } } peers { permanent_uuid: "f6720378ff514b54b38e689fd93699ef" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 34991 } }
I20260925 15:12:54.521539 13955 leader_election.cc:288] T 864b70889e434a37946e2b4040a5f1fe P 93a588223ed64f8abf3dc71226199d7c [CANDIDATE]: Term 1 election: Requested vote from peers 0d1a68cc0c8c4273abdb41ff336583c6 (127.12.8.2:42187), f6720378ff514b54b38e689fd93699ef (127.12.8.3:34991)
I20260925 15:12:54.521802 13744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "864b70889e434a37946e2b4040a5f1fe" candidate_uuid: "93a588223ed64f8abf3dc71226199d7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d1a68cc0c8c4273abdb41ff336583c6"
I20260925 15:12:54.521793 13879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "864b70889e434a37946e2b4040a5f1fe" candidate_uuid: "93a588223ed64f8abf3dc71226199d7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6720378ff514b54b38e689fd93699ef"
I20260925 15:12:54.521992 13744 raft_consensus.cc:3037] T 864b70889e434a37946e2b4040a5f1fe P 0d1a68cc0c8c4273abdb41ff336583c6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:12:54.522027 13879 raft_consensus.cc:3037] T 864b70889e434a37946e2b4040a5f1fe P f6720378ff514b54b38e689fd93699ef [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:12:54.524744 13879 raft_consensus.cc:2454] T 864b70889e434a37946e2b4040a5f1fe P f6720378ff514b54b38e689fd93699ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93a588223ed64f8abf3dc71226199d7c in term 1.
I20260925 15:12:54.524761 13744 raft_consensus.cc:2454] T 864b70889e434a37946e2b4040a5f1fe P 0d1a68cc0c8c4273abdb41ff336583c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93a588223ed64f8abf3dc71226199d7c in term 1.
I20260925 15:12:54.525120 13534 leader_election.cc:302] T 864b70889e434a37946e2b4040a5f1fe P 93a588223ed64f8abf3dc71226199d7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0d1a68cc0c8c4273abdb41ff336583c6, 93a588223ed64f8abf3dc71226199d7c, f6720378ff514b54b38e689fd93699ef; no voters:
I20260925 15:12:54.525388 13955 raft_consensus.cc:2781] T 864b70889e434a37946e2b4040a5f1fe P 93a588223ed64f8abf3dc71226199d7c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:12:54.525710 13955 raft_consensus.cc:686] T 864b70889e434a37946e2b4040a5f1fe P 93a588223ed64f8abf3dc71226199d7c [term 1 LEADER]: Becoming Leader. State: Replica: 93a588223ed64f8abf3dc71226199d7c, State: Running, Role: LEADER
I20260925 15:12:54.525965 13955 consensus_queue.cc:237] T 864b70889e434a37946e2b4040a5f1fe P 93a588223ed64f8abf3dc71226199d7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a588223ed64f8abf3dc71226199d7c" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 45655 } } peers { permanent_uuid: "0d1a68cc0c8c4273abdb41ff336583c6" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 42187 } } peers { permanent_uuid: "f6720378ff514b54b38e689fd93699ef" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 34991 } }
I20260925 15:12:54.527467 13991 tablet_replica.cc:439] T 864b70889e434a37946e2b4040a5f1fe P 93a588223ed64f8abf3dc71226199d7c: TxnStatusTablet state changed. Reason: New leader 93a588223ed64f8abf3dc71226199d7c. Latest consensus state: current_term: 1 leader_uuid: "93a588223ed64f8abf3dc71226199d7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a588223ed64f8abf3dc71226199d7c" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 45655 } } peers { permanent_uuid: "0d1a68cc0c8c4273abdb41ff336583c6" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 42187 } } peers { permanent_uuid: "f6720378ff514b54b38e689fd93699ef" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 34991 } } }
I20260925 15:12:54.527789 13991 tablet_replica.cc:442] T 864b70889e434a37946e2b4040a5f1fe P 93a588223ed64f8abf3dc71226199d7c: This TxnStatusTablet replica's current role is: LEADER
I20260925 15:12:54.528349 13993 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 15:12:54.528559 13993 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 15:12:54.528949 13444 catalog_manager.cc:5495] T 864b70889e434a37946e2b4040a5f1fe P 93a588223ed64f8abf3dc71226199d7c reported cstate change: term changed from 0 to 1, leader changed from <none> to 93a588223ed64f8abf3dc71226199d7c (127.12.8.1). New cstate: current_term: 1 leader_uuid: "93a588223ed64f8abf3dc71226199d7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a588223ed64f8abf3dc71226199d7c" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 45655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d1a68cc0c8c4273abdb41ff336583c6" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 42187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6720378ff514b54b38e689fd93699ef" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 34991 } health_report { overall_health: UNKNOWN } } }
I20260925 15:12:54.590937 13944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:54.585345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60452 (local address 127.12.8.1:45655)
0925 15:12:54.585532 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:54.585536 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:54.585608 (+ 72us) server_negotiation.cc:407] Connection header received
0925 15:12:54.585866 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:54.585869 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:54.585930 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:54.586006 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:12:54.586434 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:54.587168 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:54.589080 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:54.589663 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:54.589801 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:54.590336 (+ 535us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:12:54.590450 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:12:54.590517 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:54.590675 (+ 158us) server_negotiation.cc:299] Negotiation successful
0925 15:12:54.590729 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260925 15:12:54.591092 13987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:54.585273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.1:45655 (local address 127.0.0.1:60452)
0925 15:12:54.585515 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:54.585536 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 15:12:54.585685 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:12:54.586059 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:12:54.586064 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:12:54.586077 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 15:12:54.586301 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:54.586313 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:54.587289 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:54.587295 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:54.588957 (+ 1662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:54.588972 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:54.589835 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:54.589842 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:54.590031 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:54.590098 (+ 67us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 15:12:54.590548 (+ 450us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 15:12:54.590567 (+ 19us) client_negotiation.cc:769] Sending connection context
0925 15:12:54.590652 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 15:12:54.590791 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260925 15:12:54.595036 13879 raft_consensus.cc:1261] T 864b70889e434a37946e2b4040a5f1fe P f6720378ff514b54b38e689fd93699ef [term 1 FOLLOWER]: Refusing update from remote peer 93a588223ed64f8abf3dc71226199d7c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:12:54.595131 13744 raft_consensus.cc:1261] T 864b70889e434a37946e2b4040a5f1fe P 0d1a68cc0c8c4273abdb41ff336583c6 [term 1 FOLLOWER]: Refusing update from remote peer 93a588223ed64f8abf3dc71226199d7c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:12:54.595729 13991 consensus_queue.cc:1035] T 864b70889e434a37946e2b4040a5f1fe P 93a588223ed64f8abf3dc71226199d7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6720378ff514b54b38e689fd93699ef" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 34991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:12:54.596017 13955 consensus_queue.cc:1035] T 864b70889e434a37946e2b4040a5f1fe P 93a588223ed64f8abf3dc71226199d7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d1a68cc0c8c4273abdb41ff336583c6" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:12:54.604753 13954 tablet_replica.cc:439] T 864b70889e434a37946e2b4040a5f1fe P f6720378ff514b54b38e689fd93699ef: TxnStatusTablet state changed. Reason: New leader 93a588223ed64f8abf3dc71226199d7c. Latest consensus state: current_term: 1 leader_uuid: "93a588223ed64f8abf3dc71226199d7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a588223ed64f8abf3dc71226199d7c" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 45655 } } peers { permanent_uuid: "0d1a68cc0c8c4273abdb41ff336583c6" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 42187 } } peers { permanent_uuid: "f6720378ff514b54b38e689fd93699ef" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 34991 } } }
I20260925 15:12:54.605024 13954 tablet_replica.cc:442] T 864b70889e434a37946e2b4040a5f1fe P f6720378ff514b54b38e689fd93699ef: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:12:54.607455 13956 tablet_replica.cc:439] T 864b70889e434a37946e2b4040a5f1fe P 0d1a68cc0c8c4273abdb41ff336583c6: TxnStatusTablet state changed. Reason: New leader 93a588223ed64f8abf3dc71226199d7c. Latest consensus state: current_term: 1 leader_uuid: "93a588223ed64f8abf3dc71226199d7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a588223ed64f8abf3dc71226199d7c" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 45655 } } peers { permanent_uuid: "0d1a68cc0c8c4273abdb41ff336583c6" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 42187 } } peers { permanent_uuid: "f6720378ff514b54b38e689fd93699ef" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 34991 } } }
I20260925 15:12:54.607717 13956 tablet_replica.cc:442] T 864b70889e434a37946e2b4040a5f1fe P 0d1a68cc0c8c4273abdb41ff336583c6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:12:54.608366 13997 mvcc.cc:205] Tried to move back new op lower bound from 7333270219135365120 to 7333270218860572672. Current Snapshot: MvccSnapshot[applied={T|T < 7333270219135365120}]
I20260925 15:12:54.612149 13996 mvcc.cc:205] Tried to move back new op lower bound from 7333270219135365120 to 7333270218860572672. Current Snapshot: MvccSnapshot[applied={T|T < 7333270219135365120}]
I20260925 15:12:54.618685 13956 tablet_replica.cc:439] T 864b70889e434a37946e2b4040a5f1fe P 0d1a68cc0c8c4273abdb41ff336583c6: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "93a588223ed64f8abf3dc71226199d7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a588223ed64f8abf3dc71226199d7c" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 45655 } } peers { permanent_uuid: "0d1a68cc0c8c4273abdb41ff336583c6" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 42187 } } peers { permanent_uuid: "f6720378ff514b54b38e689fd93699ef" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 34991 } } }
I20260925 15:12:54.619082 13956 tablet_replica.cc:442] T 864b70889e434a37946e2b4040a5f1fe P 0d1a68cc0c8c4273abdb41ff336583c6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:12:54.620682 13991 tablet_replica.cc:439] T 864b70889e434a37946e2b4040a5f1fe P 93a588223ed64f8abf3dc71226199d7c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "93a588223ed64f8abf3dc71226199d7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a588223ed64f8abf3dc71226199d7c" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 45655 } } peers { permanent_uuid: "0d1a68cc0c8c4273abdb41ff336583c6" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 42187 } } peers { permanent_uuid: "f6720378ff514b54b38e689fd93699ef" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 34991 } } }
I20260925 15:12:54.620973 13991 tablet_replica.cc:442] T 864b70889e434a37946e2b4040a5f1fe P 93a588223ed64f8abf3dc71226199d7c: This TxnStatusTablet replica's current role is: LEADER
I20260925 15:12:54.622649 13954 tablet_replica.cc:439] T 864b70889e434a37946e2b4040a5f1fe P f6720378ff514b54b38e689fd93699ef: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "93a588223ed64f8abf3dc71226199d7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a588223ed64f8abf3dc71226199d7c" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 45655 } } peers { permanent_uuid: "0d1a68cc0c8c4273abdb41ff336583c6" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 42187 } } peers { permanent_uuid: "f6720378ff514b54b38e689fd93699ef" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 34991 } } }
I20260925 15:12:54.622884 13954 tablet_replica.cc:442] T 864b70889e434a37946e2b4040a5f1fe P f6720378ff514b54b38e689fd93699ef: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:12:54.624842 13991 tablet_replica.cc:439] T 864b70889e434a37946e2b4040a5f1fe P 93a588223ed64f8abf3dc71226199d7c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "93a588223ed64f8abf3dc71226199d7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a588223ed64f8abf3dc71226199d7c" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 45655 } } peers { permanent_uuid: "0d1a68cc0c8c4273abdb41ff336583c6" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 42187 } } peers { permanent_uuid: "f6720378ff514b54b38e689fd93699ef" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 34991 } } }
I20260925 15:12:54.625087 13991 tablet_replica.cc:442] T 864b70889e434a37946e2b4040a5f1fe P 93a588223ed64f8abf3dc71226199d7c: This TxnStatusTablet replica's current role is: LEADER
I20260925 15:12:54.636564 14004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:54.629278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41930 (local address 127.12.8.2:42187)
0925 15:12:54.630690 (+ 1412us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:54.630694 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:54.630708 (+ 14us) server_negotiation.cc:407] Connection header received
0925 15:12:54.630778 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:54.630781 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:54.630923 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:54.631077 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:12:54.631610 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:54.632741 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:54.634060 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:54.634724 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:54.634848 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:54.635217 (+ 369us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:12:54.635931 (+ 714us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:12:54.636070 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:54.636124 (+ 54us) server_negotiation.cc:299] Negotiation successful
0925 15:12:54.636204 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1301,"thread_start_us":1196,"threads_started":1}
I20260925 15:12:54.651665 13944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:54.646140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.2:40645 (local address 127.12.8.1:45655)
0925 15:12:54.646329 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:54.646334 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:54.646798 (+ 464us) server_negotiation.cc:407] Connection header received
0925 15:12:54.646999 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:54.647005 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:54.647076 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:54.647173 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:12:54.647652 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:54.648422 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:54.649779 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:54.650305 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:54.650453 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:54.650819 (+ 366us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:12:54.650939 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:12:54.651031 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:54.651238 (+ 207us) server_negotiation.cc:299] Negotiation successful
0925 15:12:54.651308 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 15:12:54.651744 14008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:54.646138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.1:45655 (local address 127.12.8.2:40645)
0925 15:12:54.646697 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:54.646712 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:12:54.646860 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:12:54.647200 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:12:54.647205 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:12:54.647216 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 15:12:54.647512 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:54.647519 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:54.648554 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:54.648562 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:54.649644 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:54.649655 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:54.650473 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:54.650479 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:54.650607 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:54.650653 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 15:12:54.651032 (+ 379us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 15:12:54.651045 (+ 13us) client_negotiation.cc:769] Sending connection context
0925 15:12:54.651205 (+ 160us) client_negotiation.cc:240] Negotiation successful
0925 15:12:54.651361 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":330,"threads_started":1}
I20260925 15:12:54.660676 13879 raft_consensus.cc:1261] T fea290f10c34443f8c9727d62bb12375 P f6720378ff514b54b38e689fd93699ef [term 1 FOLLOWER]: Refusing update from remote peer 0d1a68cc0c8c4273abdb41ff336583c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:12:54.661507 13606 raft_consensus.cc:1261] T fea290f10c34443f8c9727d62bb12375 P 93a588223ed64f8abf3dc71226199d7c [term 1 FOLLOWER]: Refusing update from remote peer 0d1a68cc0c8c4273abdb41ff336583c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:12:54.661360 13979 consensus_queue.cc:1035] T fea290f10c34443f8c9727d62bb12375 P 0d1a68cc0c8c4273abdb41ff336583c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6720378ff514b54b38e689fd93699ef" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 34991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:12:54.662567 13979 consensus_queue.cc:1035] T fea290f10c34443f8c9727d62bb12375 P 0d1a68cc0c8c4273abdb41ff336583c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93a588223ed64f8abf3dc71226199d7c" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 45655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:12:54.714602 14004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:54.707347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.1:45467 (local address 127.12.8.2:42187)
0925 15:12:54.707515 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:54.707521 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:54.708304 (+ 783us) server_negotiation.cc:407] Connection header received
0925 15:12:54.708591 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:54.708598 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:54.708679 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:54.708743 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:12:54.709812 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:54.710806 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:54.712517 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:54.713113 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:54.713216 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:54.713848 (+ 632us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:12:54.713963 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:12:54.714011 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:54.714238 (+ 227us) server_negotiation.cc:299] Negotiation successful
0925 15:12:54.714309 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 15:12:54.714813 14012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:54.707423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.2:42187 (local address 127.12.8.1:45467)
0925 15:12:54.708195 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:54.708210 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:12:54.708419 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:12:54.708955 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:12:54.708959 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:12:54.708971 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 15:12:54.709362 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:54.709374 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:54.710967 (+ 1593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:54.710970 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:54.712355 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:54.712369 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:54.713384 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:54.713390 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:54.713582 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:54.713653 (+ 71us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 15:12:54.714074 (+ 421us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 15:12:54.714080 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 15:12:54.714210 (+ 130us) client_negotiation.cc:240] Negotiation successful
0925 15:12:54.714340 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":407,"threads_started":1}
I20260925 15:12:54.762634 13444 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38736:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:12:54.762897 13444 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38736:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:12:54.766387 13444 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6676713ee26a4eb188da557ac038061e: Sending DeleteTablet for 3 replicas of tablet fea290f10c34443f8c9727d62bb12375
I20260925 15:12:54.767210 13720 tablet_service.cc:1514] Processing DeleteTablet for tablet fea290f10c34443f8c9727d62bb12375 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:12:54 UTC) from {username='kudu', principal='kudu/127.12.8.62@KRBTEST.COM'} at 127.0.0.1:41926
I20260925 15:12:54.767896 14017 tablet_replica.cc:330] T fea290f10c34443f8c9727d62bb12375 P 0d1a68cc0c8c4273abdb41ff336583c6: stopping tablet replica
I20260925 15:12:54.767915 13859 tablet_service.cc:1514] Processing DeleteTablet for tablet fea290f10c34443f8c9727d62bb12375 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:12:54 UTC) from {username='kudu', principal='kudu/127.12.8.62@KRBTEST.COM'} at 127.0.0.1:41780
I20260925 15:12:54.768097 14017 raft_consensus.cc:2229] T fea290f10c34443f8c9727d62bb12375 P 0d1a68cc0c8c4273abdb41ff336583c6 [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:12:54.768428 13586 tablet_service.cc:1514] Processing DeleteTablet for tablet fea290f10c34443f8c9727d62bb12375 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:12:54 UTC) from {username='kudu', principal='kudu/127.12.8.62@KRBTEST.COM'} at 127.0.0.1:60428
I20260925 15:12:54.768638 14017 raft_consensus.cc:2258] T fea290f10c34443f8c9727d62bb12375 P 0d1a68cc0c8c4273abdb41ff336583c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:12:54.769699 14017 ts_tablet_manager.cc:1905] T fea290f10c34443f8c9727d62bb12375 P 0d1a68cc0c8c4273abdb41ff336583c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:12:54.776954 14017 ts_tablet_manager.cc:1918] T fea290f10c34443f8c9727d62bb12375 P 0d1a68cc0c8c4273abdb41ff336583c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 15:12:54.777097 14017 log.cc:1196] T fea290f10c34443f8c9727d62bb12375 P 0d1a68cc0c8c4273abdb41ff336583c6: Deleting WAL directory at /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/minicluster-data/ts-1/wal/wals/fea290f10c34443f8c9727d62bb12375
I20260925 15:12:54.777563 14017 ts_tablet_manager.cc:1939] T fea290f10c34443f8c9727d62bb12375 P 0d1a68cc0c8c4273abdb41ff336583c6: Deleting consensus metadata
I20260925 15:12:54.778625 13429 catalog_manager.cc:4841] TS 0d1a68cc0c8c4273abdb41ff336583c6 (127.12.8.2:42187): tablet fea290f10c34443f8c9727d62bb12375 (table test-table [id=6e30f4f916f94be7bf9b15a9a2721067]) successfully deleted
I20260925 15:12:54.782967 14019 tablet_replica.cc:330] T fea290f10c34443f8c9727d62bb12375 P 93a588223ed64f8abf3dc71226199d7c: stopping tablet replica
I20260925 15:12:54.783178 14019 raft_consensus.cc:2229] T fea290f10c34443f8c9727d62bb12375 P 93a588223ed64f8abf3dc71226199d7c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:12:54.783460 14019 raft_consensus.cc:2258] T fea290f10c34443f8c9727d62bb12375 P 93a588223ed64f8abf3dc71226199d7c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:12:54.785027 14018 tablet_replica.cc:330] T fea290f10c34443f8c9727d62bb12375 P f6720378ff514b54b38e689fd93699ef: stopping tablet replica
I20260925 15:12:54.785224 14018 raft_consensus.cc:2229] T fea290f10c34443f8c9727d62bb12375 P f6720378ff514b54b38e689fd93699ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:12:54.785440 14018 raft_consensus.cc:2258] T fea290f10c34443f8c9727d62bb12375 P f6720378ff514b54b38e689fd93699ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:12:54.786571 14019 ts_tablet_manager.cc:1905] T fea290f10c34443f8c9727d62bb12375 P 93a588223ed64f8abf3dc71226199d7c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:12:54.786926 14018 ts_tablet_manager.cc:1905] T fea290f10c34443f8c9727d62bb12375 P f6720378ff514b54b38e689fd93699ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:12:54.794302 14019 ts_tablet_manager.cc:1918] T fea290f10c34443f8c9727d62bb12375 P 93a588223ed64f8abf3dc71226199d7c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 15:12:54.794445 14019 log.cc:1196] T fea290f10c34443f8c9727d62bb12375 P 93a588223ed64f8abf3dc71226199d7c: Deleting WAL directory at /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/minicluster-data/ts-0/wal/wals/fea290f10c34443f8c9727d62bb12375
I20260925 15:12:54.794945 14019 ts_tablet_manager.cc:1939] T fea290f10c34443f8c9727d62bb12375 P 93a588223ed64f8abf3dc71226199d7c: Deleting consensus metadata
I20260925 15:12:54.796085 13428 catalog_manager.cc:4841] TS 93a588223ed64f8abf3dc71226199d7c (127.12.8.1:45655): tablet fea290f10c34443f8c9727d62bb12375 (table test-table [id=6e30f4f916f94be7bf9b15a9a2721067]) successfully deleted
I20260925 15:12:54.796648 14018 ts_tablet_manager.cc:1918] T fea290f10c34443f8c9727d62bb12375 P f6720378ff514b54b38e689fd93699ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 15:12:54.796783 14018 log.cc:1196] T fea290f10c34443f8c9727d62bb12375 P f6720378ff514b54b38e689fd93699ef: Deleting WAL directory at /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349162370577-12320-0/minicluster-data/ts-2/wal/wals/fea290f10c34443f8c9727d62bb12375
I20260925 15:12:54.797206 14018 ts_tablet_manager.cc:1939] T fea290f10c34443f8c9727d62bb12375 P f6720378ff514b54b38e689fd93699ef: Deleting consensus metadata
I20260925 15:12:54.798262 13431 catalog_manager.cc:4841] TS f6720378ff514b54b38e689fd93699ef (127.12.8.3:34991): tablet fea290f10c34443f8c9727d62bb12375 (table test-table [id=6e30f4f916f94be7bf9b15a9a2721067]) successfully deleted
Sep 25 15:12:54 dist-test-slave-p468 krb5kdc[13399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349173, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.8.1@KRBTEST.COM
I20260925 15:12:54.814944 13944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:54.784037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60460 (local address 127.12.8.1:45655)
0925 15:12:54.784216 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:54.784222 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:54.795089 (+ 10867us) server_negotiation.cc:407] Connection header received
0925 15:12:54.795600 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:54.795606 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:54.795683 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:54.795814 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:54.798345 (+ 2531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:54.799499 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:54.801448 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:54.802153 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:54.802280 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:54.807500 (+ 5220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:54.807520 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:54.807539 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:12:54.807578 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:54.811578 (+ 4000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:54.812413 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:54.812425 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:54.812430 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:54.812586 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:54.813009 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:54.813015 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:54.813018 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:54.813476 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:54.813792 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:54.814099 (+ 307us) server_negotiation.cc:299] Negotiation successful
0925 15:12:54.814384 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260925 15:12:54.815734 13646 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60460
I20260925 15:12:54.854139 14024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:54.819235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38756 (local address 127.12.8.62:39285)
0925 15:12:54.834687 (+ 15452us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:54.834695 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:54.838618 (+ 3923us) server_negotiation.cc:407] Connection header received
0925 15:12:54.838856 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:54.838862 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:54.838963 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:54.839087 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:54.840782 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:54.841932 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:54.843807 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:54.844507 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:54.844638 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:54.847613 (+ 2975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:54.847621 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:54.847626 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:12:54.847668 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:54.851174 (+ 3506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:54.851990 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:54.851997 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:54.852003 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:54.852146 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:54.852573 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:54.852578 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:54.852582 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:54.852929 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:54.853173 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:54.853524 (+ 351us) server_negotiation.cc:299] Negotiation successful
0925 15:12:54.853757 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15307,"thread_start_us":15460,"threads_started":1}
W20260925 15:12:54.854955 13444 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38756
I20260925 15:12:54.857239 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 13515
I20260925 15:12:54.878325 13944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:54.859951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60464 (local address 127.12.8.1:45655)
0925 15:12:54.860144 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:54.860150 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:54.863003 (+ 2853us) server_negotiation.cc:407] Connection header received
0925 15:12:54.863138 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:54.863143 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:54.863221 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:54.863324 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:54.864793 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:54.865998 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:54.867769 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:54.868481 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:54.868617 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:54.871433 (+ 2816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:54.871439 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:54.871444 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:12:54.871495 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:54.875280 (+ 3785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:54.876086 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:54.876092 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:54.876096 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:54.876242 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:54.876654 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:54.876660 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:54.876664 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:54.877020 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:54.877280 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:54.877614 (+ 334us) server_negotiation.cc:299] Negotiation successful
0925 15:12:54.877844 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260925 15:12:54.879045 13646 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60464
W20260925 15:12:54.879617 12320 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 13515: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 15:12:54.879802 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 13515
I20260925 15:12:54.904863 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 13655
Sep 25 15:12:54 dist-test-slave-p468 krb5kdc[13399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349173, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.8.2@KRBTEST.COM
I20260925 15:12:54.922253 14004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:54.905706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41942 (local address 127.12.8.2:42187)
0925 15:12:54.905890 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:54.905894 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:54.905913 (+ 19us) server_negotiation.cc:407] Connection header received
0925 15:12:54.906123 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:54.906127 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:54.906235 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:54.906337 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:54.907970 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:54.908711 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:54.910734 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:54.911167 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:54.911386 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:54.916355 (+ 4969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:54.916369 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:54.916384 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:12:54.916431 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:54.919488 (+ 3057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:54.920163 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:54.920175 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:54.920178 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:54.920329 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:54.920694 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:54.920700 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:54.920703 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:54.921056 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:54.921361 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:54.921657 (+ 296us) server_negotiation.cc:299] Negotiation successful
0925 15:12:54.921873 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260925 15:12:54.923005 13785 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41942
W20260925 15:12:54.923661 12320 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 13655: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 15:12:54.923840 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 13655
I20260925 15:12:54.943336 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 13794
Sep 25 15:12:54 dist-test-slave-p468 krb5kdc[13399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349173, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.8.3@KRBTEST.COM
I20260925 15:12:54.958775 14026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:54.944150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41794 (local address 127.12.8.3:34991)
0925 15:12:54.944645 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:54.944650 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:54.944665 (+ 15us) server_negotiation.cc:407] Connection header received
0925 15:12:54.944717 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:54.944721 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:54.944777 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:54.944909 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:54.946286 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:54.946966 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:54.948236 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:54.948655 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:54.948743 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:54.953166 (+ 4423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:54.953181 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:54.953195 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:12:54.953234 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:54.956015 (+ 2781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:54.956658 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:54.956665 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:54.956667 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:54.956783 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:54.957220 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:54.957223 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:54.957225 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:54.957546 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:54.957757 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:54.958090 (+ 333us) server_negotiation.cc:299] Negotiation successful
0925 15:12:54.958310 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":289,"threads_started":1}
W20260925 15:12:54.959514 13923 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41794
W20260925 15:12:54.960230 12320 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 13794: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 15:12:54.960405 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 13794
I20260925 15:12:54.976533 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 13412
I20260925 15:12:54.977650 13484 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:12:55.091378 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 13412
2026-09-25T15:12:55Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4690 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:12:55 dist-test-slave-p468 krb5kdc[14030](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:12:55 dist-test-slave-p468 krb5kdc[14030](info): set up 2 sockets
Sep 25 15:12:55 dist-test-slave-p468 krb5kdc[14030](info): commencing operation
krb5kdc: starting...
W20260925 15:12:56.182176 12320 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.004s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/krb5kdc/test-user.keytab.
Sep 25 15:12:56 dist-test-slave-p468 krb5kdc[14030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349176, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:12:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:12:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.8.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab.
Entry for principal kudu/127.12.8.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab.
I20260925 15:12:56.363585 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.62:33113
--webserver_interface=127.12.8.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab
--principal=kudu/127.12.8.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:39115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.62:33113
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:12:56.674724 14043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:12:56.675078 14043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:12:56.675247 14043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:12:56.682427 14043 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:12:56.682566 14043 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 15:12:56.682655 14043 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:12:56.682775 14043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:12:56.682888 14043 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:12:56.682979 14043 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:12:56.692978 14043 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:39115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.8.62:33113
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab
--principal=kudu/127.12.8.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.62:33113
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.8.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:12:56.694284 14043 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:12:56.695604 14043 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:12:56.707599 14048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:12:56.707564 14049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:12:56.707540 14051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:12:56.708544 14043 server_base.cc:1034] running on GCE node
I20260925 15:12:56.709254 14043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:12:56.710588 14043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:12:56.711825 14043 hybrid_clock.cc:648] HybridClock initialized: now 1790349176711758 us; error 43 us; skew 500 ppm
I20260925 15:12:56.712188 14043 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:12:56 dist-test-slave-p468 krb5kdc[14030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349176, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:12:56.715251 14043 init.cc:377] Logged in from keytab as kudu/127.12.8.62@KRBTEST.COM (short username kudu)
I20260925 15:12:56.716686 14043 webserver.cc:460] Webserver started at http://127.12.8.62:35117/ using document root <none> and password file <none>
I20260925 15:12:56.717357 14043 fs_manager.cc:359] Metadata directory not provided
I20260925 15:12:56.717486 14043 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:12:56.717784 14043 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:12:56.720640 14043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/minicluster-data/master-0/data/instance:
uuid: "8ee449b6924246da81d3c6ac299728f6"
format_stamp: "Formatted at 2026-09-25 15:12:56 on dist-test-slave-p468"
server_key: "7cf1300365bd8d07f97fc85b24b42229"
server_key_iv: "775847e5ad760b084e15fcfc09bcfa7f"
server_key_version: "encryptionkey@0"
I20260925 15:12:56.721396 14043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/minicluster-data/master-0/wal/instance:
uuid: "8ee449b6924246da81d3c6ac299728f6"
format_stamp: "Formatted at 2026-09-25 15:12:56 on dist-test-slave-p468"
server_key: "7cf1300365bd8d07f97fc85b24b42229"
server_key_iv: "775847e5ad760b084e15fcfc09bcfa7f"
server_key_version: "encryptionkey@0"
I20260925 15:12:56.726282 14043 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260925 15:12:56.729573 14058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:12:56.730921 14043 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:12:56.731092 14043 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/minicluster-data/master-0/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/minicluster-data/master-0/wal
uuid: "8ee449b6924246da81d3c6ac299728f6"
format_stamp: "Formatted at 2026-09-25 15:12:56 on dist-test-slave-p468"
server_key: "7cf1300365bd8d07f97fc85b24b42229"
server_key_iv: "775847e5ad760b084e15fcfc09bcfa7f"
server_key_version: "encryptionkey@0"
I20260925 15:12:56.731295 14043 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:12:56.752897 14043 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:12:56.756839 14043 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:12:56.757107 14043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:12:56.777833 14043 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.62:33113
I20260925 15:12:56.777885 14119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.62:33113 every 8 connection(s)
I20260925 15:12:56.779124 14043 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
I20260925 15:12:56.779806 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 14043
I20260925 15:12:56.780015 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/minicluster-data/master-0/wal/instance
I20260925 15:12:56.780862 12320 external_mini_cluster.cc:1428] Setting key 56db1a294f97a72dd355e2710e9e0803
I20260925 15:12:56.783560 14120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:12:56.795397 14120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ee449b6924246da81d3c6ac299728f6: Bootstrap starting.
Sep 25 15:12:56 dist-test-slave-p468 krb5kdc[14030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349176, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
I20260925 15:12:56.799180 14120 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ee449b6924246da81d3c6ac299728f6: Neither blocks nor log segments found. Creating new log.
I20260925 15:12:56.800443 14120 log.cc:824] T 00000000000000000000000000000000 P 8ee449b6924246da81d3c6ac299728f6: Log is configured to *not* fsync() on all Append() calls
I20260925 15:12:56.803184 14120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ee449b6924246da81d3c6ac299728f6: No bootstrap required, opened a new log
I20260925 15:12:56.804191 14123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:56.782830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42512 (local address 127.12.8.62:33113)
0925 15:12:56.783607 (+ 777us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:56.783624 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:56.783673 (+ 49us) server_negotiation.cc:407] Connection header received
0925 15:12:56.785856 (+ 2183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:56.785893 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:56.786404 (+ 511us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:56.786808 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:56.788088 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:56.790126 (+ 2038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:56.791436 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:56.792481 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:56.792654 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:56.796420 (+ 3766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:56.796429 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:56.796446 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:12:56.796495 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:56.800350 (+ 3855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:56.801082 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:56.801089 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:56.801095 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:56.801255 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:56.801640 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:56.801644 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:56.801647 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:56.802119 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:56.802410 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:56.802843 (+ 433us) server_negotiation.cc:299] Negotiation successful
0925 15:12:56.803254 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":321,"threads_started":1}
I20260925 15:12:56.811508 14120 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ee449b6924246da81d3c6ac299728f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ee449b6924246da81d3c6ac299728f6" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 33113 } }
I20260925 15:12:56.811942 14120 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ee449b6924246da81d3c6ac299728f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:12:56.812057 14120 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ee449b6924246da81d3c6ac299728f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ee449b6924246da81d3c6ac299728f6, State: Initialized, Role: FOLLOWER
I20260925 15:12:56.812609 14120 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ee449b6924246da81d3c6ac299728f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ee449b6924246da81d3c6ac299728f6" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 33113 } }
I20260925 15:12:56.812842 14120 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ee449b6924246da81d3c6ac299728f6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:12:56.812999 14120 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ee449b6924246da81d3c6ac299728f6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:12:56.813174 14120 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ee449b6924246da81d3c6ac299728f6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:12:56.815589 14120 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ee449b6924246da81d3c6ac299728f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ee449b6924246da81d3c6ac299728f6" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 33113 } }
I20260925 15:12:56.816097 14120 leader_election.cc:302] T 00000000000000000000000000000000 P 8ee449b6924246da81d3c6ac299728f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ee449b6924246da81d3c6ac299728f6; no voters:
I20260925 15:12:56.816682 14120 leader_election.cc:288] T 00000000000000000000000000000000 P 8ee449b6924246da81d3c6ac299728f6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:12:56.816773 14125 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ee449b6924246da81d3c6ac299728f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:12:56.817467 14125 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ee449b6924246da81d3c6ac299728f6 [term 1 LEADER]: Becoming Leader. State: Replica: 8ee449b6924246da81d3c6ac299728f6, State: Running, Role: LEADER
I20260925 15:12:56.817997 14125 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ee449b6924246da81d3c6ac299728f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ee449b6924246da81d3c6ac299728f6" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 33113 } }
I20260925 15:12:56.818756 14120 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ee449b6924246da81d3c6ac299728f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:12:56.820986 14127 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ee449b6924246da81d3c6ac299728f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ee449b6924246da81d3c6ac299728f6. Latest consensus state: current_term: 1 leader_uuid: "8ee449b6924246da81d3c6ac299728f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ee449b6924246da81d3c6ac299728f6" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 33113 } } }
I20260925 15:12:56.821429 14127 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ee449b6924246da81d3c6ac299728f6 [sys.catalog]: This master's current role is: LEADER
I20260925 15:12:56.821185 14126 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ee449b6924246da81d3c6ac299728f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8ee449b6924246da81d3c6ac299728f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ee449b6924246da81d3c6ac299728f6" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 33113 } } }
I20260925 15:12:56.821552 14126 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ee449b6924246da81d3c6ac299728f6 [sys.catalog]: This master's current role is: LEADER
I20260925 15:12:56.826648 14130 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:12:56.831776 14130 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:12:56.840458 14130 catalog_manager.cc:1347] Generated new cluster ID: f236d31df8f74e9f9b92a3c385e2a92b
I20260925 15:12:56.840569 14130 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:12:56.854295 14130 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:12:56.855479 14130 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:12:56.865203 14130 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8ee449b6924246da81d3c6ac299728f6: Generated new TSK 0
I20260925 15:12:56.865957 14130 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.8.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.1.keytab.
Entry for principal kudu/127.12.8.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.1.keytab.
I20260925 15:12:56.922017 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.1:0
--local_ip_for_outbound_sockets=127.12.8.1
--webserver_interface=127.12.8.1
--webserver_port=0
--tserver_master_addrs=127.12.8.62:33113
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.1.keytab
--principal=kudu/127.12.8.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:39115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:12:57.242570 14146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:12:57.242841 14146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:12:57.242942 14146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:12:57.249961 14146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:12:57.250070 14146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:12:57.250154 14146 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 15:12:57.250260 14146 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 15:12:57.250402 14146 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.1
I20260925 15:12:57.260174 14146 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:39115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.1.keytab
--principal=kudu/127.12.8.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.8.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.8.62:33113
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.1
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:12:57.261526 14146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:12:57.262929 14146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:12:57.276093 14152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:12:57.276161 14151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:12:57.276135 14154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:12:57.276954 14146 server_base.cc:1034] running on GCE node
I20260925 15:12:57.277494 14146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:12:57.278463 14146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:12:57.279654 14146 hybrid_clock.cc:648] HybridClock initialized: now 1790349177279592 us; error 59 us; skew 500 ppm
I20260925 15:12:57.279963 14146 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:12:57 dist-test-slave-p468 krb5kdc[14030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349177, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:12:57.283015 14146 init.cc:377] Logged in from keytab as kudu/127.12.8.1@KRBTEST.COM (short username kudu)
I20260925 15:12:57.284374 14146 webserver.cc:460] Webserver started at http://127.12.8.1:36473/ using document root <none> and password file <none>
I20260925 15:12:57.284991 14146 fs_manager.cc:359] Metadata directory not provided
I20260925 15:12:57.285086 14146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:12:57.285324 14146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:12:57.288060 14146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/minicluster-data/ts-0/data/instance:
uuid: "60ceefa28bab42e0adcb2b6345c27a62"
format_stamp: "Formatted at 2026-09-25 15:12:57 on dist-test-slave-p468"
server_key: "94baf4816523d69f8da6088d10556b45"
server_key_iv: "c6921d312e58d1e14d8cab73002ad8c1"
server_key_version: "encryptionkey@0"
I20260925 15:12:57.288746 14146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/minicluster-data/ts-0/wal/instance:
uuid: "60ceefa28bab42e0adcb2b6345c27a62"
format_stamp: "Formatted at 2026-09-25 15:12:57 on dist-test-slave-p468"
server_key: "94baf4816523d69f8da6088d10556b45"
server_key_iv: "c6921d312e58d1e14d8cab73002ad8c1"
server_key_version: "encryptionkey@0"
I20260925 15:12:57.293507 14146 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 15:12:57.296792 14161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:12:57.298121 14146 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 15:12:57.298331 14146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/minicluster-data/ts-0/wal
uuid: "60ceefa28bab42e0adcb2b6345c27a62"
format_stamp: "Formatted at 2026-09-25 15:12:57 on dist-test-slave-p468"
server_key: "94baf4816523d69f8da6088d10556b45"
server_key_iv: "c6921d312e58d1e14d8cab73002ad8c1"
server_key_version: "encryptionkey@0"
I20260925 15:12:57.298513 14146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:12:57.337299 14146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:12:57.341652 14146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:12:57.341958 14146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:12:57.344446 14146 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:12:57.344595 14146 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:12:57.344729 14146 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:12:57.344838 14146 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 15:12:57 dist-test-slave-p468 krb5kdc[14030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349177, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.1@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
I20260925 15:12:57.369177 14170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:57.345240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.1:40933 (local address 127.12.8.62:33113)
0925 15:12:57.345782 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:57.345788 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:57.346989 (+ 1201us) server_negotiation.cc:407] Connection header received
0925 15:12:57.349016 (+ 2027us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:57.349022 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:57.349109 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:57.349227 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:57.352069 (+ 2842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:57.353083 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:57.355395 (+ 2312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:57.355980 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:57.356146 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:57.361418 (+ 5272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:57.361423 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:57.361426 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:12:57.361458 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:57.365358 (+ 3900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:57.366657 (+ 1299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:57.366664 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:57.366667 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:57.366790 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:57.367348 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:57.367363 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:57.367365 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:57.367626 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:57.367734 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:57.368761 (+ 1027us) server_negotiation.cc:299] Negotiation successful
0925 15:12:57.368919 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":314,"threads_started":1}
I20260925 15:12:57.371474 14171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:57.345909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:33113 (local address 127.12.8.1:40933)
0925 15:12:57.346804 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:57.346860 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 15:12:57.348728 (+ 1868us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:12:57.349715 (+ 987us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:12:57.349731 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:12:57.350792 (+ 1061us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:12:57.351827 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:57.351859 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:57.353238 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:57.353243 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:57.355241 (+ 1998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:57.355252 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:57.356299 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:57.356313 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:57.356695 (+ 382us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:57.357774 (+ 1079us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:12:57.357814 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:12:57.361104 (+ 3290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:12:57.365573 (+ 4469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:12:57.365587 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:12:57.365609 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:12:57.366499 (+ 890us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:12:57.366907 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:12:57.366914 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:12:57.366921 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:12:57.367234 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:12:57.367755 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:12:57.367766 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:12:57.368317 (+ 551us) client_negotiation.cc:769] Sending connection context
0925 15:12:57.368693 (+ 376us) client_negotiation.cc:240] Negotiation successful
0925 15:12:57.369382 (+ 689us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"thread_start_us":426,"threads_started":1}
I20260925 15:12:57.396852 14170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:57.380863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.1:38555 (local address 127.12.8.62:33113)
0925 15:12:57.381074 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:57.381081 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:57.381712 (+ 631us) server_negotiation.cc:407] Connection header received
0925 15:12:57.381919 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:57.381922 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:57.381980 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:57.382057 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:57.383704 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:57.384653 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:57.387283 (+ 2630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:57.388109 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:57.388264 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:57.390029 (+ 1765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:57.390038 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:57.390046 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:12:57.390093 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:57.394020 (+ 3927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:57.394955 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:57.394961 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:57.394964 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:57.395097 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:57.395450 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:57.395455 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:57.395465 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:57.395723 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:57.395875 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:57.396263 (+ 388us) server_negotiation.cc:299] Negotiation successful
0925 15:12:57.396500 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 15:12:57.396950 14251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:57.380862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:33113 (local address 127.12.8.1:38555)
0925 15:12:57.381583 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:57.381614 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0925 15:12:57.381799 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:12:57.382122 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:12:57.382129 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:12:57.382947 (+ 818us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:12:57.383524 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:57.383546 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:57.384803 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:57.384810 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:57.387055 (+ 2245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:57.387077 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:57.388346 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:57.388353 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:57.388517 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:57.389151 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:12:57.389167 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:12:57.389839 (+ 672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:12:57.394171 (+ 4332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:12:57.394175 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:12:57.394178 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:12:57.394789 (+ 611us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:12:57.395207 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:12:57.395213 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:12:57.395216 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:12:57.395334 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:12:57.395836 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:12:57.395841 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:12:57.396017 (+ 176us) client_negotiation.cc:769] Sending connection context
0925 15:12:57.396148 (+ 131us) client_negotiation.cc:240] Negotiation successful
0925 15:12:57.396467 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":363,"threads_started":1}
I20260925 15:12:57.400353 14146 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.1:34225
I20260925 15:12:57.400399 14281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.1:34225 every 8 connection(s)
I20260925 15:12:57.401726 14146 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/minicluster-data/ts-0/data/info.pb
I20260925 15:12:57.403784 14282 heartbeater.cc:347] Connected to a master server at 127.12.8.62:33113
I20260925 15:12:57.404012 14282 heartbeater.cc:464] Registering TS with master...
I20260925 15:12:57.404610 14282 heartbeater.cc:511] Master 127.12.8.62:33113 requested a full tablet report, sending...
I20260925 15:12:57.406625 14075 ts_manager.cc:194] Registered new tserver with Master: 60ceefa28bab42e0adcb2b6345c27a62 (127.12.8.1:34225)
I20260925 15:12:57.408077 14075 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.1@KRBTEST.COM'} at 127.12.8.1:40933
I20260925 15:12:57.411401 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 14146
I20260925 15:12:57.411566 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/minicluster-data/ts-0/wal/instance
I20260925 15:12:57.412187 12320 external_mini_cluster.cc:1428] Setting key be90deab4f09fcb5a78c22a73a7f416f
WARNING: no policy specified for kudu/127.12.8.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.2.keytab.
Entry for principal kudu/127.12.8.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.2.keytab.
I20260925 15:12:57.454111 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.2:0
--local_ip_for_outbound_sockets=127.12.8.2
--webserver_interface=127.12.8.2
--webserver_port=0
--tserver_master_addrs=127.12.8.62:33113
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.2.keytab
--principal=kudu/127.12.8.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:39115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:12:57.767716 14286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:12:57.768062 14286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:12:57.768252 14286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:12:57.776293 14286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:12:57.776450 14286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:12:57.776611 14286 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 15:12:57.776729 14286 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 15:12:57.776868 14286 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.2
I20260925 15:12:57.787371 14286 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:39115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.2.keytab
--principal=kudu/127.12.8.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.8.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.8.62:33113
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.2
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:12:57.788599 14286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:12:57.789917 14286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:12:57.803363 14291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:12:57.803319 14292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:12:57.803854 14294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:12:57.804420 14286 server_base.cc:1034] running on GCE node
I20260925 15:12:57.804998 14286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:12:57.805866 14286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:12:57.807088 14286 hybrid_clock.cc:648] HybridClock initialized: now 1790349177807052 us; error 48 us; skew 500 ppm
I20260925 15:12:57.807399 14286 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:12:57 dist-test-slave-p468 krb5kdc[14030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349177, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:12:57.810415 14286 init.cc:377] Logged in from keytab as kudu/127.12.8.2@KRBTEST.COM (short username kudu)
I20260925 15:12:57.811815 14286 webserver.cc:460] Webserver started at http://127.12.8.2:39275/ using document root <none> and password file <none>
I20260925 15:12:57.812472 14286 fs_manager.cc:359] Metadata directory not provided
I20260925 15:12:57.812613 14286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:12:57.812916 14286 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:12:57.815717 14286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/minicluster-data/ts-1/data/instance:
uuid: "4037cbe8f1de462ca703b3580eeb1af5"
format_stamp: "Formatted at 2026-09-25 15:12:57 on dist-test-slave-p468"
server_key: "41dd70d7f59ca1112b7096b834f68c63"
server_key_iv: "54b0c935fbca994df4b0e005995677af"
server_key_version: "encryptionkey@0"
I20260925 15:12:57.816442 14286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/minicluster-data/ts-1/wal/instance:
uuid: "4037cbe8f1de462ca703b3580eeb1af5"
format_stamp: "Formatted at 2026-09-25 15:12:57 on dist-test-slave-p468"
server_key: "41dd70d7f59ca1112b7096b834f68c63"
server_key_iv: "54b0c935fbca994df4b0e005995677af"
server_key_version: "encryptionkey@0"
I20260925 15:12:57.821337 14286 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 15:12:57.824599 14301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:12:57.825922 14286 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 15:12:57.826107 14286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/minicluster-data/ts-1/wal
uuid: "4037cbe8f1de462ca703b3580eeb1af5"
format_stamp: "Formatted at 2026-09-25 15:12:57 on dist-test-slave-p468"
server_key: "41dd70d7f59ca1112b7096b834f68c63"
server_key_iv: "54b0c935fbca994df4b0e005995677af"
server_key_version: "encryptionkey@0"
I20260925 15:12:57.826361 14286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:12:57.877324 14286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:12:57.881722 14286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:12:57.882037 14286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:12:57.884848 14286 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:12:57.884989 14286 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:12:57.885175 14286 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:12:57.885284 14286 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 15:12:57 dist-test-slave-p468 krb5kdc[14030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349177, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.2@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
I20260925 15:12:57.907989 14170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:57.885625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.2:51569 (local address 127.12.8.62:33113)
0925 15:12:57.885885 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:57.885894 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:57.887471 (+ 1577us) server_negotiation.cc:407] Connection header received
0925 15:12:57.889471 (+ 2000us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:57.889476 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:57.889571 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:57.889682 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:57.891935 (+ 2253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:57.893366 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:57.895890 (+ 2524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:57.896516 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:57.896672 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:57.901660 (+ 4988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:57.901669 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:57.901676 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:12:57.901721 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:57.904844 (+ 3123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:57.905570 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:57.905575 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:57.905579 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:57.905713 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:57.906148 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:57.906153 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:57.906156 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:57.906499 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:57.906657 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:57.907376 (+ 719us) server_negotiation.cc:299] Negotiation successful
0925 15:12:57.907628 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
I20260925 15:12:57.909049 14310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:57.886422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:33113 (local address 127.12.8.2:51569)
0925 15:12:57.887301 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:57.887340 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 15:12:57.889196 (+ 1856us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:12:57.890001 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:12:57.890019 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:12:57.890812 (+ 793us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:12:57.891677 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:57.891696 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:57.893561 (+ 1865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:57.893566 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:57.895731 (+ 2165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:57.895747 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:57.896775 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:57.896788 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:57.897025 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:57.897870 (+ 845us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:12:57.897892 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:12:57.901335 (+ 3443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:12:57.904991 (+ 3656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:12:57.905000 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:12:57.905018 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:12:57.905440 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:12:57.905825 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:12:57.905831 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:12:57.905835 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:12:57.906028 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:12:57.906648 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:12:57.906656 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:12:57.907015 (+ 359us) client_negotiation.cc:769] Sending connection context
0925 15:12:57.907285 (+ 270us) client_negotiation.cc:240] Negotiation successful
0925 15:12:57.907872 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":635,"thread_start_us":412,"threads_started":1}
I20260925 15:12:57.931540 14170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:57.916562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.2:34147 (local address 127.12.8.62:33113)
0925 15:12:57.916753 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:57.916759 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:57.916973 (+ 214us) server_negotiation.cc:407] Connection header received
0925 15:12:57.917171 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:57.917175 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:57.917254 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:57.917351 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:57.919250 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:57.920411 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:57.921788 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:57.922268 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:57.922404 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:57.924373 (+ 1969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:57.924378 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:57.924382 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:12:57.924412 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:57.928232 (+ 3820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:57.929106 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:57.929114 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:57.929119 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:57.929248 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:57.929667 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:57.929674 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:57.929679 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:57.930128 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:57.930329 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:57.930966 (+ 637us) server_negotiation.cc:299] Negotiation successful
0925 15:12:57.931199 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 15:12:57.931674 14360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:57.915933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:33113 (local address 127.12.8.2:34147)
0925 15:12:57.916836 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:57.916865 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0925 15:12:57.917038 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:12:57.917451 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:12:57.917458 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:12:57.918442 (+ 984us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:12:57.919077 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:57.919101 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:57.920565 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:57.920572 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:57.921642 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:57.921654 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:57.922453 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:57.922459 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:57.922666 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:57.923427 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:12:57.923444 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:12:57.924165 (+ 721us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:12:57.928366 (+ 4201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:12:57.928370 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:12:57.928373 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:12:57.928939 (+ 566us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:12:57.929364 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:12:57.929370 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:12:57.929373 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:12:57.929544 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:12:57.930397 (+ 853us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:12:57.930407 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:12:57.930680 (+ 273us) client_negotiation.cc:769] Sending connection context
0925 15:12:57.930884 (+ 204us) client_negotiation.cc:240] Negotiation successful
0925 15:12:57.931212 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":689,"thread_start_us":711,"threads_started":1}
I20260925 15:12:57.947621 14286 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.2:43891
I20260925 15:12:57.947660 14420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.2:43891 every 8 connection(s)
I20260925 15:12:57.948877 14286 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/minicluster-data/ts-1/data/info.pb
I20260925 15:12:57.951052 14421 heartbeater.cc:347] Connected to a master server at 127.12.8.62:33113
I20260925 15:12:57.951305 14421 heartbeater.cc:464] Registering TS with master...
I20260925 15:12:57.951550 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 14286
I20260925 15:12:57.951757 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/minicluster-data/ts-1/wal/instance
I20260925 15:12:57.952021 14421 heartbeater.cc:511] Master 127.12.8.62:33113 requested a full tablet report, sending...
I20260925 15:12:57.952286 12320 external_mini_cluster.cc:1428] Setting key 6bf75afddfb68b3b015abc921edca649
I20260925 15:12:57.953420 14075 ts_manager.cc:194] Registered new tserver with Master: 4037cbe8f1de462ca703b3580eeb1af5 (127.12.8.2:43891)
I20260925 15:12:57.954465 14075 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.2@KRBTEST.COM'} at 127.12.8.2:51569
WARNING: no policy specified for kudu/127.12.8.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.3.keytab.
Entry for principal kudu/127.12.8.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.3.keytab.
I20260925 15:12:57.994069 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.3:0
--local_ip_for_outbound_sockets=127.12.8.3
--webserver_interface=127.12.8.3
--webserver_port=0
--tserver_master_addrs=127.12.8.62:33113
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.3.keytab
--principal=kudu/127.12.8.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:39115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:12:58.309015 14425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:12:58.309280 14425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:12:58.309440 14425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:12:58.316478 14425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:12:58.316577 14425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:12:58.316723 14425 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 15:12:58.316782 14425 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 15:12:58.316891 14425 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.3
I20260925 15:12:58.327363 14425 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:39115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.3.keytab
--principal=kudu/127.12.8.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.8.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.8.62:33113
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.3
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:12:58.328680 14425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:12:58.330087 14425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:12:58.342878 14430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:12:58.342933 14433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:12:58.342959 14431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:12:58.344170 14425 server_base.cc:1034] running on GCE node
I20260925 15:12:58.344774 14425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:12:58.345659 14425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:12:58.346920 14425 hybrid_clock.cc:648] HybridClock initialized: now 1790349178346852 us; error 55 us; skew 500 ppm
I20260925 15:12:58.347229 14425 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:12:58 dist-test-slave-p468 krb5kdc[14030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349178, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:12:58.355336 14425 init.cc:377] Logged in from keytab as kudu/127.12.8.3@KRBTEST.COM (short username kudu)
I20260925 15:12:58.356909 14425 webserver.cc:460] Webserver started at http://127.12.8.3:42623/ using document root <none> and password file <none>
I20260925 15:12:58.357657 14425 fs_manager.cc:359] Metadata directory not provided
I20260925 15:12:58.357815 14425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:12:58.358156 14425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:12:58.361455 14425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/minicluster-data/ts-2/data/instance:
uuid: "3e607a2588f04ef7bd2c71fb7752782d"
format_stamp: "Formatted at 2026-09-25 15:12:58 on dist-test-slave-p468"
server_key: "aeb74907a67edf6b33a1efca1e54204f"
server_key_iv: "c05e5d0981973382a64ce6a2ae9a6a49"
server_key_version: "encryptionkey@0"
I20260925 15:12:58.362313 14425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/minicluster-data/ts-2/wal/instance:
uuid: "3e607a2588f04ef7bd2c71fb7752782d"
format_stamp: "Formatted at 2026-09-25 15:12:58 on dist-test-slave-p468"
server_key: "aeb74907a67edf6b33a1efca1e54204f"
server_key_iv: "c05e5d0981973382a64ce6a2ae9a6a49"
server_key_version: "encryptionkey@0"
I20260925 15:12:58.367699 14425 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260925 15:12:58.371433 14440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:12:58.372910 14425 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 15:12:58.373100 14425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/minicluster-data/ts-2/wal
uuid: "3e607a2588f04ef7bd2c71fb7752782d"
format_stamp: "Formatted at 2026-09-25 15:12:58 on dist-test-slave-p468"
server_key: "aeb74907a67edf6b33a1efca1e54204f"
server_key_iv: "c05e5d0981973382a64ce6a2ae9a6a49"
server_key_version: "encryptionkey@0"
I20260925 15:12:58.373318 14425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:12:58.390070 14425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:12:58.394426 14425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:12:58.394658 14425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:12:58.397472 14425 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:12:58.397620 14425 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:12:58.397774 14425 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:12:58.397886 14425 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:12:58.410656 14282 heartbeater.cc:503] Master 127.12.8.62:33113 was elected leader, sending a full tablet report...
Sep 25 15:12:58 dist-test-slave-p468 krb5kdc[14030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349178, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.3@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
I20260925 15:12:58.422109 14170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:58.398033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.3:60397 (local address 127.12.8.62:33113)
0925 15:12:58.398312 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:58.398321 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:58.399755 (+ 1434us) server_negotiation.cc:407] Connection header received
0925 15:12:58.402266 (+ 2511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:58.402272 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:58.402398 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:58.402508 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:58.404988 (+ 2480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:58.406259 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:58.408060 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:58.408851 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:58.409005 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:58.413991 (+ 4986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:58.414000 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:58.414008 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:12:58.414057 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:58.418440 (+ 4383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:58.419376 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:58.419383 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:58.419387 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:58.419539 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:58.420205 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:58.420211 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:58.420217 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:58.420666 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:58.420837 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:58.421533 (+ 696us) server_negotiation.cc:299] Negotiation successful
0925 15:12:58.421748 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 15:12:58.423079 14449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:58.398711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:33113 (local address 127.12.8.3:60397)
0925 15:12:58.399585 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:58.399636 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 15:12:58.401976 (+ 2340us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:12:58.402917 (+ 941us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:12:58.402933 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:12:58.403776 (+ 843us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:12:58.404742 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:58.404773 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:58.406484 (+ 1711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:58.406491 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:58.407886 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:58.407907 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:58.409148 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:58.409157 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:58.409433 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:58.410274 (+ 841us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:12:58.410324 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:12:58.413755 (+ 3431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:12:58.418561 (+ 4806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:12:58.418568 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:12:58.418603 (+ 35us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:12:58.419188 (+ 585us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:12:58.419706 (+ 518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:12:58.419710 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:12:58.419714 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:12:58.420073 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:12:58.420817 (+ 744us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:12:58.420828 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:12:58.421153 (+ 325us) client_negotiation.cc:769] Sending connection context
0925 15:12:58.421422 (+ 269us) client_negotiation.cc:240] Negotiation successful
0925 15:12:58.421918 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":413,"threads_started":1}
I20260925 15:12:58.446177 14170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:58.429483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.3:40831 (local address 127.12.8.62:33113)
0925 15:12:58.429659 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:58.429665 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:58.429887 (+ 222us) server_negotiation.cc:407] Connection header received
0925 15:12:58.430163 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:58.430168 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:58.430276 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:58.430373 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:58.432047 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:58.433166 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:58.435774 (+ 2608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:58.436202 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:58.436300 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:58.439018 (+ 2718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:58.439027 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:58.439035 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:12:58.439081 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:58.442746 (+ 3665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:58.443611 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:58.443617 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:58.443622 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:58.443777 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:58.444283 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:58.444292 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:58.444296 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:58.444611 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:58.444810 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:58.445498 (+ 688us) server_negotiation.cc:299] Negotiation successful
0925 15:12:58.445786 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 15:12:58.446265 14510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:58.428991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:33113 (local address 127.12.8.3:40831)
0925 15:12:58.429777 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:58.429802 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 15:12:58.429998 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:12:58.430594 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:12:58.430601 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:12:58.431306 (+ 705us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:12:58.431813 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:58.431828 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:58.433321 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:58.433338 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:58.435607 (+ 2269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:58.435639 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:58.436408 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:58.436417 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:58.436646 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:58.437387 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:12:58.437418 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:12:58.438790 (+ 1372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:12:58.442910 (+ 4120us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:12:58.442921 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:12:58.442931 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:12:58.443466 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:12:58.443940 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:12:58.443948 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:12:58.443952 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:12:58.444138 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:12:58.444890 (+ 752us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:12:58.444897 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:12:58.445164 (+ 267us) client_negotiation.cc:769] Sending connection context
0925 15:12:58.445363 (+ 199us) client_negotiation.cc:240] Negotiation successful
0925 15:12:58.445708 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":443,"threads_started":1}
I20260925 15:12:58.454878 14425 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.3:43695
I20260925 15:12:58.454921 14559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.3:43695 every 8 connection(s)
I20260925 15:12:58.456055 14425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/minicluster-data/ts-2/data/info.pb
I20260925 15:12:58.457952 14560 heartbeater.cc:347] Connected to a master server at 127.12.8.62:33113
I20260925 15:12:58.458169 14560 heartbeater.cc:464] Registering TS with master...
I20260925 15:12:58.459096 14560 heartbeater.cc:511] Master 127.12.8.62:33113 requested a full tablet report, sending...
I20260925 15:12:58.460454 14075 ts_manager.cc:194] Registered new tserver with Master: 3e607a2588f04ef7bd2c71fb7752782d (127.12.8.3:43695)
I20260925 15:12:58.461246 14075 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.3@KRBTEST.COM'} at 127.12.8.3:60397
I20260925 15:12:58.461591 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 14425
I20260925 15:12:58.461745 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/minicluster-data/ts-2/wal/instance
I20260925 15:12:58.462196 12320 external_mini_cluster.cc:1428] Setting key 849d632d8c54f541198bc5e0347e0a65
I20260925 15:12:58.465188 12320 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 15:12:58 dist-test-slave-p468 krb5kdc[14030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349178, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 15:12:58 dist-test-slave-p468 krb5kdc[14030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349178, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
I20260925 15:12:58.512383 14170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:58.497820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42528 (local address 127.12.8.62:33113)
0925 15:12:58.497991 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:58.497996 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:58.498295 (+ 299us) server_negotiation.cc:407] Connection header received
0925 15:12:58.498455 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:58.498460 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:58.498526 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:58.498621 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:58.499870 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:58.500671 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:58.501943 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:58.502492 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:58.502589 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:58.507117 (+ 4528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:58.507122 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:58.507125 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:12:58.507161 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:58.509993 (+ 2832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:58.510587 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:58.510591 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:58.510593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:58.510760 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:58.511076 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:58.511079 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:58.511081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:58.511344 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:58.511529 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:58.511852 (+ 323us) server_negotiation.cc:299] Negotiation successful
0925 15:12:58.512024 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 15:12:58.516126 14075 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42528:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:12:58.523005 14075 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:12:58.551759 14577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:58.535876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49170 (local address 127.12.8.2:43891)
0925 15:12:58.544093 (+ 8217us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:58.544105 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:58.544146 (+ 41us) server_negotiation.cc:407] Connection header received
0925 15:12:58.544259 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:58.544267 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:58.544518 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:58.544719 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:12:58.545371 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:58.546606 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:58.549324 (+ 2718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:58.550498 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:58.550633 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:58.550852 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:58.551296 (+ 444us) server_negotiation.cc:299] Negotiation successful
0925 15:12:58.551467 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8052,"thread_start_us":7959,"threads_started":1}
I20260925 15:12:58.552001 14573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:58.534710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.1:34225 (local address 127.0.0.1:48958)
0925 15:12:58.535598 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:58.535686 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0925 15:12:58.535849 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:12:58.545335 (+ 9486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:12:58.545341 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:12:58.545368 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:12:58.545836 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:58.545852 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:58.547644 (+ 1792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:58.547649 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:58.549381 (+ 1732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:58.549394 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:58.551083 (+ 1689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:58.551090 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:58.551255 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:58.551287 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 15:12:58.551391 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 15:12:58.551528 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406,"mutex_wait_us":60,"spinlock_wait_cycles":3840,"thread_start_us":394,"threads_started":1}
I20260925 15:12:58.551712 14576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:58.535238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.2:43891 (local address 127.0.0.1:49170)
0925 15:12:58.543796 (+ 8558us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:58.543817 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 15:12:58.543931 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:12:58.544771 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:12:58.544781 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:12:58.544813 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:12:58.545180 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:58.545192 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:58.546766 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:58.546771 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:58.549162 (+ 2391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:58.549175 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:58.550695 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:58.550701 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:58.550901 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:58.550968 (+ 67us) client_negotiation.cc:769] Sending connection context
0925 15:12:58.551096 (+ 128us) client_negotiation.cc:240] Negotiation successful
0925 15:12:58.551287 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8420,"thread_start_us":8904,"threads_started":1}
I20260925 15:12:58.552280 14574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:58.534791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48958 (local address 127.12.8.1:34225)
0925 15:12:58.544761 (+ 9970us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:58.544768 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:58.544791 (+ 23us) server_negotiation.cc:407] Connection header received
0925 15:12:58.544888 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:58.544895 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:58.545126 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:58.545340 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:12:58.546014 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:58.547490 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:58.549538 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:58.550816 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:58.550988 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:58.551189 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:58.551811 (+ 622us) server_negotiation.cc:299] Negotiation successful
0925 15:12:58.551988 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9835,"thread_start_us":9703,"threads_started":1}
I20260925 15:12:58.553613 14578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:58.536321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60456 (local address 127.12.8.3:43695)
0925 15:12:58.538960 (+ 2639us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:58.538971 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:58.544498 (+ 5527us) server_negotiation.cc:407] Connection header received
0925 15:12:58.544779 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:58.544786 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:58.544992 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:58.545105 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:12:58.546143 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:58.547641 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:58.550090 (+ 2449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:58.552614 (+ 2524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:58.552765 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:58.552978 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:58.553137 (+ 159us) server_negotiation.cc:299] Negotiation successful
0925 15:12:58.553294 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2427,"thread_start_us":2059,"threads_started":1}
I20260925 15:12:58.556157 14212 tablet_service.cc:1467] Processing CreateTablet for tablet 10867a35e08f46f0801bd1fc11103f2c (DEFAULT_TABLE table=test-table [id=e0d8da9bc2eb4a6b89a0924652d8f8ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:12:58.556157 14355 tablet_service.cc:1467] Processing CreateTablet for tablet 10867a35e08f46f0801bd1fc11103f2c (DEFAULT_TABLE table=test-table [id=e0d8da9bc2eb4a6b89a0924652d8f8ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:12:58.557091 14575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:58.535847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.3:43695 (local address 127.0.0.1:60456)
0925 15:12:58.544400 (+ 8553us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:58.544415 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:12:58.544591 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:12:58.545386 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:12:58.545391 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:12:58.545420 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:12:58.545887 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:58.545896 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:58.547783 (+ 1887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:58.547788 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:58.549922 (+ 2134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:58.549935 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:58.552802 (+ 2867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:58.552808 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:58.552968 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:58.552998 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 15:12:58.553088 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 15:12:58.553212 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8446,"mutex_wait_us":23,"spinlock_wait_cycles":4864,"thread_start_us":7673,"threads_started":1}
I20260925 15:12:58.557561 14212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10867a35e08f46f0801bd1fc11103f2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:12:58.557572 14355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10867a35e08f46f0801bd1fc11103f2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:12:58.560410 14490 tablet_service.cc:1467] Processing CreateTablet for tablet 10867a35e08f46f0801bd1fc11103f2c (DEFAULT_TABLE table=test-table [id=e0d8da9bc2eb4a6b89a0924652d8f8ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:12:58.561545 14490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10867a35e08f46f0801bd1fc11103f2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:12:58.571853 14580 tablet_bootstrap.cc:492] T 10867a35e08f46f0801bd1fc11103f2c P 4037cbe8f1de462ca703b3580eeb1af5: Bootstrap starting.
I20260925 15:12:58.573536 14581 tablet_bootstrap.cc:492] T 10867a35e08f46f0801bd1fc11103f2c P 3e607a2588f04ef7bd2c71fb7752782d: Bootstrap starting.
I20260925 15:12:58.574338 14579 tablet_bootstrap.cc:492] T 10867a35e08f46f0801bd1fc11103f2c P 60ceefa28bab42e0adcb2b6345c27a62: Bootstrap starting.
I20260925 15:12:58.576437 14581 tablet_bootstrap.cc:654] T 10867a35e08f46f0801bd1fc11103f2c P 3e607a2588f04ef7bd2c71fb7752782d: Neither blocks nor log segments found. Creating new log.
I20260925 15:12:58.576929 14579 tablet_bootstrap.cc:654] T 10867a35e08f46f0801bd1fc11103f2c P 60ceefa28bab42e0adcb2b6345c27a62: Neither blocks nor log segments found. Creating new log.
I20260925 15:12:58.577637 14581 log.cc:824] T 10867a35e08f46f0801bd1fc11103f2c P 3e607a2588f04ef7bd2c71fb7752782d: Log is configured to *not* fsync() on all Append() calls
I20260925 15:12:58.578245 14579 log.cc:824] T 10867a35e08f46f0801bd1fc11103f2c P 60ceefa28bab42e0adcb2b6345c27a62: Log is configured to *not* fsync() on all Append() calls
I20260925 15:12:58.579229 14580 tablet_bootstrap.cc:654] T 10867a35e08f46f0801bd1fc11103f2c P 4037cbe8f1de462ca703b3580eeb1af5: Neither blocks nor log segments found. Creating new log.
I20260925 15:12:58.580379 14580 log.cc:824] T 10867a35e08f46f0801bd1fc11103f2c P 4037cbe8f1de462ca703b3580eeb1af5: Log is configured to *not* fsync() on all Append() calls
I20260925 15:12:58.580760 14581 tablet_bootstrap.cc:492] T 10867a35e08f46f0801bd1fc11103f2c P 3e607a2588f04ef7bd2c71fb7752782d: No bootstrap required, opened a new log
I20260925 15:12:58.581092 14581 ts_tablet_manager.cc:1397] T 10867a35e08f46f0801bd1fc11103f2c P 3e607a2588f04ef7bd2c71fb7752782d: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20260925 15:12:58.582006 14579 tablet_bootstrap.cc:492] T 10867a35e08f46f0801bd1fc11103f2c P 60ceefa28bab42e0adcb2b6345c27a62: No bootstrap required, opened a new log
I20260925 15:12:58.582316 14579 ts_tablet_manager.cc:1397] T 10867a35e08f46f0801bd1fc11103f2c P 60ceefa28bab42e0adcb2b6345c27a62: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260925 15:12:58.583137 14580 tablet_bootstrap.cc:492] T 10867a35e08f46f0801bd1fc11103f2c P 4037cbe8f1de462ca703b3580eeb1af5: No bootstrap required, opened a new log
I20260925 15:12:58.583396 14580 ts_tablet_manager.cc:1397] T 10867a35e08f46f0801bd1fc11103f2c P 4037cbe8f1de462ca703b3580eeb1af5: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.006s
I20260925 15:12:58.589174 14579 raft_consensus.cc:348] T 10867a35e08f46f0801bd1fc11103f2c P 60ceefa28bab42e0adcb2b6345c27a62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ceefa28bab42e0adcb2b6345c27a62" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 34225 } } peers { permanent_uuid: "3e607a2588f04ef7bd2c71fb7752782d" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43695 } } peers { permanent_uuid: "4037cbe8f1de462ca703b3580eeb1af5" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 43891 } }
I20260925 15:12:58.589509 14579 raft_consensus.cc:374] T 10867a35e08f46f0801bd1fc11103f2c P 60ceefa28bab42e0adcb2b6345c27a62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:12:58.589635 14579 raft_consensus.cc:729] T 10867a35e08f46f0801bd1fc11103f2c P 60ceefa28bab42e0adcb2b6345c27a62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ceefa28bab42e0adcb2b6345c27a62, State: Initialized, Role: FOLLOWER
I20260925 15:12:58.590233 14579 consensus_queue.cc:260] T 10867a35e08f46f0801bd1fc11103f2c P 60ceefa28bab42e0adcb2b6345c27a62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ceefa28bab42e0adcb2b6345c27a62" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 34225 } } peers { permanent_uuid: "3e607a2588f04ef7bd2c71fb7752782d" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43695 } } peers { permanent_uuid: "4037cbe8f1de462ca703b3580eeb1af5" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 43891 } }
I20260925 15:12:58.591537 14581 raft_consensus.cc:348] T 10867a35e08f46f0801bd1fc11103f2c P 3e607a2588f04ef7bd2c71fb7752782d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ceefa28bab42e0adcb2b6345c27a62" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 34225 } } peers { permanent_uuid: "3e607a2588f04ef7bd2c71fb7752782d" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43695 } } peers { permanent_uuid: "4037cbe8f1de462ca703b3580eeb1af5" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 43891 } }
I20260925 15:12:58.591900 14579 ts_tablet_manager.cc:1428] T 10867a35e08f46f0801bd1fc11103f2c P 60ceefa28bab42e0adcb2b6345c27a62: Time spent starting tablet: real 0.009s user 0.005s sys 0.004s
I20260925 15:12:58.591993 14581 raft_consensus.cc:374] T 10867a35e08f46f0801bd1fc11103f2c P 3e607a2588f04ef7bd2c71fb7752782d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:12:58.592137 14581 raft_consensus.cc:729] T 10867a35e08f46f0801bd1fc11103f2c P 3e607a2588f04ef7bd2c71fb7752782d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e607a2588f04ef7bd2c71fb7752782d, State: Initialized, Role: FOLLOWER
I20260925 15:12:58.592729 14581 consensus_queue.cc:260] T 10867a35e08f46f0801bd1fc11103f2c P 3e607a2588f04ef7bd2c71fb7752782d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ceefa28bab42e0adcb2b6345c27a62" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 34225 } } peers { permanent_uuid: "3e607a2588f04ef7bd2c71fb7752782d" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43695 } } peers { permanent_uuid: "4037cbe8f1de462ca703b3580eeb1af5" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 43891 } }
I20260925 15:12:58.593679 14580 raft_consensus.cc:348] T 10867a35e08f46f0801bd1fc11103f2c P 4037cbe8f1de462ca703b3580eeb1af5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ceefa28bab42e0adcb2b6345c27a62" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 34225 } } peers { permanent_uuid: "3e607a2588f04ef7bd2c71fb7752782d" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43695 } } peers { permanent_uuid: "4037cbe8f1de462ca703b3580eeb1af5" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 43891 } }
I20260925 15:12:58.594070 14580 raft_consensus.cc:374] T 10867a35e08f46f0801bd1fc11103f2c P 4037cbe8f1de462ca703b3580eeb1af5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:12:58.594242 14580 raft_consensus.cc:729] T 10867a35e08f46f0801bd1fc11103f2c P 4037cbe8f1de462ca703b3580eeb1af5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4037cbe8f1de462ca703b3580eeb1af5, State: Initialized, Role: FOLLOWER
I20260925 15:12:58.594430 14581 ts_tablet_manager.cc:1428] T 10867a35e08f46f0801bd1fc11103f2c P 3e607a2588f04ef7bd2c71fb7752782d: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260925 15:12:58.594847 14580 consensus_queue.cc:260] T 10867a35e08f46f0801bd1fc11103f2c P 4037cbe8f1de462ca703b3580eeb1af5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ceefa28bab42e0adcb2b6345c27a62" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 34225 } } peers { permanent_uuid: "3e607a2588f04ef7bd2c71fb7752782d" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43695 } } peers { permanent_uuid: "4037cbe8f1de462ca703b3580eeb1af5" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 43891 } }
I20260925 15:12:58.595114 14560 heartbeater.cc:503] Master 127.12.8.62:33113 was elected leader, sending a full tablet report...
I20260925 15:12:58.596161 14421 heartbeater.cc:503] Master 127.12.8.62:33113 was elected leader, sending a full tablet report...
I20260925 15:12:58.596501 14580 ts_tablet_manager.cc:1428] T 10867a35e08f46f0801bd1fc11103f2c P 4037cbe8f1de462ca703b3580eeb1af5: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
I20260925 15:12:58.620142 14587 raft_consensus.cc:482] T 10867a35e08f46f0801bd1fc11103f2c P 4037cbe8f1de462ca703b3580eeb1af5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:12:58.620432 14587 raft_consensus.cc:504] T 10867a35e08f46f0801bd1fc11103f2c P 4037cbe8f1de462ca703b3580eeb1af5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ceefa28bab42e0adcb2b6345c27a62" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 34225 } } peers { permanent_uuid: "3e607a2588f04ef7bd2c71fb7752782d" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43695 } } peers { permanent_uuid: "4037cbe8f1de462ca703b3580eeb1af5" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 43891 } }
I20260925 15:12:58.622051 14587 leader_election.cc:288] T 10867a35e08f46f0801bd1fc11103f2c P 4037cbe8f1de462ca703b3580eeb1af5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60ceefa28bab42e0adcb2b6345c27a62 (127.12.8.1:34225), 3e607a2588f04ef7bd2c71fb7752782d (127.12.8.3:43695)
I20260925 15:12:58.630941 14574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:58.622561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.2:52999 (local address 127.12.8.1:34225)
0925 15:12:58.622761 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:58.622766 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:58.623204 (+ 438us) server_negotiation.cc:407] Connection header received
0925 15:12:58.623403 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:58.623408 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:58.623506 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:58.623612 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:12:58.624351 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:58.625523 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:58.628144 (+ 2621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:58.629511 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:58.629675 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:58.629792 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:58.630522 (+ 730us) server_negotiation.cc:299] Negotiation successful
0925 15:12:58.630667 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 15:12:58.630946 14578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:58.623426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.2:50673 (local address 127.12.8.3:43695)
0925 15:12:58.623688 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:58.623697 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:58.623973 (+ 276us) server_negotiation.cc:407] Connection header received
0925 15:12:58.624120 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:58.624124 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:58.624244 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:58.624362 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:12:58.624901 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:58.625927 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:58.628246 (+ 2319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:58.629570 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:58.629761 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:58.629878 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:58.630522 (+ 644us) server_negotiation.cc:299] Negotiation successful
0925 15:12:58.630706 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 15:12:58.631074 14589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:58.623184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.3:43695 (local address 127.12.8.2:50673)
0925 15:12:58.623857 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:58.623879 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 15:12:58.624005 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:12:58.624396 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:12:58.624400 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:12:58.624420 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:12:58.624744 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:58.624751 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:58.626047 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:58.626053 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:58.628098 (+ 2045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:58.628110 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:58.629874 (+ 1764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:58.629879 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:58.630262 (+ 383us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:58.630382 (+ 120us) client_negotiation.cc:769] Sending connection context
0925 15:12:58.630477 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 15:12:58.630687 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"mutex_wait_us":156,"thread_start_us":342,"threads_started":1}
I20260925 15:12:58.631326 14588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:58.622446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.1:34225 (local address 127.12.8.2:52999)
0925 15:12:58.623096 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:58.623115 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 15:12:58.623246 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:12:58.623758 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:12:58.623764 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:12:58.623792 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:12:58.624200 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:58.624209 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:58.625687 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:58.625694 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:58.627961 (+ 2267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:58.627977 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:58.629869 (+ 1892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:58.629875 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:58.630258 (+ 383us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:58.630383 (+ 125us) client_negotiation.cc:769] Sending connection context
0925 15:12:58.630477 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 15:12:58.630664 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"mutex_wait_us":158,"thread_start_us":338,"threads_started":1}
I20260925 15:12:58.632179 14515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10867a35e08f46f0801bd1fc11103f2c" candidate_uuid: "4037cbe8f1de462ca703b3580eeb1af5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e607a2588f04ef7bd2c71fb7752782d" is_pre_election: true
I20260925 15:12:58.632280 14232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10867a35e08f46f0801bd1fc11103f2c" candidate_uuid: "4037cbe8f1de462ca703b3580eeb1af5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ceefa28bab42e0adcb2b6345c27a62" is_pre_election: true
I20260925 15:12:58.632834 14515 raft_consensus.cc:2454] T 10867a35e08f46f0801bd1fc11103f2c P 3e607a2588f04ef7bd2c71fb7752782d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4037cbe8f1de462ca703b3580eeb1af5 in term 0.
I20260925 15:12:58.632849 14232 raft_consensus.cc:2454] T 10867a35e08f46f0801bd1fc11103f2c P 60ceefa28bab42e0adcb2b6345c27a62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4037cbe8f1de462ca703b3580eeb1af5 in term 0.
I20260925 15:12:58.633474 14303 leader_election.cc:302] T 10867a35e08f46f0801bd1fc11103f2c P 4037cbe8f1de462ca703b3580eeb1af5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e607a2588f04ef7bd2c71fb7752782d, 4037cbe8f1de462ca703b3580eeb1af5; no voters:
I20260925 15:12:58.633869 14587 raft_consensus.cc:2781] T 10867a35e08f46f0801bd1fc11103f2c P 4037cbe8f1de462ca703b3580eeb1af5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:12:58.634018 14587 raft_consensus.cc:482] T 10867a35e08f46f0801bd1fc11103f2c P 4037cbe8f1de462ca703b3580eeb1af5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:12:58.634124 14587 raft_consensus.cc:3037] T 10867a35e08f46f0801bd1fc11103f2c P 4037cbe8f1de462ca703b3580eeb1af5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:12:58.636673 14587 raft_consensus.cc:504] T 10867a35e08f46f0801bd1fc11103f2c P 4037cbe8f1de462ca703b3580eeb1af5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ceefa28bab42e0adcb2b6345c27a62" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 34225 } } peers { permanent_uuid: "3e607a2588f04ef7bd2c71fb7752782d" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43695 } } peers { permanent_uuid: "4037cbe8f1de462ca703b3580eeb1af5" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 43891 } }
I20260925 15:12:58.637203 14587 leader_election.cc:288] T 10867a35e08f46f0801bd1fc11103f2c P 4037cbe8f1de462ca703b3580eeb1af5 [CANDIDATE]: Term 1 election: Requested vote from peers 60ceefa28bab42e0adcb2b6345c27a62 (127.12.8.1:34225), 3e607a2588f04ef7bd2c71fb7752782d (127.12.8.3:43695)
I20260925 15:12:58.637621 14232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10867a35e08f46f0801bd1fc11103f2c" candidate_uuid: "4037cbe8f1de462ca703b3580eeb1af5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ceefa28bab42e0adcb2b6345c27a62"
I20260925 15:12:58.637662 14515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10867a35e08f46f0801bd1fc11103f2c" candidate_uuid: "4037cbe8f1de462ca703b3580eeb1af5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e607a2588f04ef7bd2c71fb7752782d"
I20260925 15:12:58.637830 14232 raft_consensus.cc:3037] T 10867a35e08f46f0801bd1fc11103f2c P 60ceefa28bab42e0adcb2b6345c27a62 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:12:58.637835 14515 raft_consensus.cc:3037] T 10867a35e08f46f0801bd1fc11103f2c P 3e607a2588f04ef7bd2c71fb7752782d [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:12:58.640013 14515 raft_consensus.cc:2454] T 10867a35e08f46f0801bd1fc11103f2c P 3e607a2588f04ef7bd2c71fb7752782d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4037cbe8f1de462ca703b3580eeb1af5 in term 1.
I20260925 15:12:58.640231 14232 raft_consensus.cc:2454] T 10867a35e08f46f0801bd1fc11103f2c P 60ceefa28bab42e0adcb2b6345c27a62 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4037cbe8f1de462ca703b3580eeb1af5 in term 1.
I20260925 15:12:58.640388 14303 leader_election.cc:302] T 10867a35e08f46f0801bd1fc11103f2c P 4037cbe8f1de462ca703b3580eeb1af5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e607a2588f04ef7bd2c71fb7752782d, 4037cbe8f1de462ca703b3580eeb1af5; no voters:
I20260925 15:12:58.640702 14587 raft_consensus.cc:2781] T 10867a35e08f46f0801bd1fc11103f2c P 4037cbe8f1de462ca703b3580eeb1af5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:12:58.641144 14587 raft_consensus.cc:686] T 10867a35e08f46f0801bd1fc11103f2c P 4037cbe8f1de462ca703b3580eeb1af5 [term 1 LEADER]: Becoming Leader. State: Replica: 4037cbe8f1de462ca703b3580eeb1af5, State: Running, Role: LEADER
I20260925 15:12:58.641619 14587 consensus_queue.cc:237] T 10867a35e08f46f0801bd1fc11103f2c P 4037cbe8f1de462ca703b3580eeb1af5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ceefa28bab42e0adcb2b6345c27a62" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 34225 } } peers { permanent_uuid: "3e607a2588f04ef7bd2c71fb7752782d" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43695 } } peers { permanent_uuid: "4037cbe8f1de462ca703b3580eeb1af5" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 43891 } }
I20260925 15:12:58.646343 14074 catalog_manager.cc:5495] T 10867a35e08f46f0801bd1fc11103f2c P 4037cbe8f1de462ca703b3580eeb1af5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4037cbe8f1de462ca703b3580eeb1af5 (127.12.8.2). New cstate: current_term: 1 leader_uuid: "4037cbe8f1de462ca703b3580eeb1af5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ceefa28bab42e0adcb2b6345c27a62" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 34225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e607a2588f04ef7bd2c71fb7752782d" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4037cbe8f1de462ca703b3580eeb1af5" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 43891 } health_report { overall_health: HEALTHY } } }
W20260925 15:12:58.654047 14283 tablet.cc:2369] T 10867a35e08f46f0801bd1fc11103f2c P 60ceefa28bab42e0adcb2b6345c27a62: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 15:12:58 dist-test-slave-p468 krb5kdc[14030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349176, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.62@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
I20260925 15:12:58.696724 14597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:58.684400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:33113 (local address 127.0.0.1:42530)
0925 15:12:58.684926 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:58.684941 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:12:58.685067 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:12:58.685344 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:12:58.685347 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:12:58.685993 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:12:58.686359 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:58.686369 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:58.687324 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:58.687329 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:58.688367 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:58.688379 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:58.689037 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:58.689041 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:58.689144 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:58.689819 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:12:58.689835 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:12:58.691953 (+ 2118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:12:58.694521 (+ 2568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:12:58.694529 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:12:58.694535 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:12:58.694978 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:12:58.695302 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:12:58.695320 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:12:58.695322 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:12:58.695481 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:12:58.695949 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:12:58.695955 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:12:58.696093 (+ 138us) client_negotiation.cc:769] Sending connection context
0925 15:12:58.696225 (+ 132us) client_negotiation.cc:240] Negotiation successful
0925 15:12:58.696401 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":291,"threads_started":1}
I20260925 15:12:58.696753 14170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:58.684380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42530 (local address 127.12.8.62:33113)
0925 15:12:58.684538 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:58.684545 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:58.685022 (+ 477us) server_negotiation.cc:407] Connection header received
0925 15:12:58.685163 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:58.685167 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:58.685223 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:58.685301 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:58.686507 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:58.687189 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:58.688489 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:58.688886 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:58.688971 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:58.692115 (+ 3144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:58.692119 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:58.692122 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:12:58.692149 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:58.694410 (+ 2261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:58.695082 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:58.695086 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:58.695088 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:58.695211 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:58.695569 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:58.695572 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:58.695574 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:58.695816 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:58.695922 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:58.696302 (+ 380us) server_negotiation.cc:299] Negotiation successful
0925 15:12:58.696513 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":49}
W20260925 15:12:58.700902 14422 tablet.cc:2369] T 10867a35e08f46f0801bd1fc11103f2c P 4037cbe8f1de462ca703b3580eeb1af5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:12:58.701299 14074 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.12.8.62@KRBTEST.COM'} at 127.0.0.1:42530:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 15:12:58.702464 14074 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260925 15:12:58.707536 14561 tablet.cc:2369] T 10867a35e08f46f0801bd1fc11103f2c P 3e607a2588f04ef7bd2c71fb7752782d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:12:58.708891 14355 tablet_service.cc:1467] Processing CreateTablet for tablet f9a010986c824dc58eb9a83b027fd9c2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ac90ff506e76437080d91910582b00aa]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 15:12:58.708815 14212 tablet_service.cc:1467] Processing CreateTablet for tablet f9a010986c824dc58eb9a83b027fd9c2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ac90ff506e76437080d91910582b00aa]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 15:12:58.709702 14355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9a010986c824dc58eb9a83b027fd9c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:12:58.709741 14212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9a010986c824dc58eb9a83b027fd9c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:12:58.709293 14490 tablet_service.cc:1467] Processing CreateTablet for tablet f9a010986c824dc58eb9a83b027fd9c2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ac90ff506e76437080d91910582b00aa]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 15:12:58.710045 14490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9a010986c824dc58eb9a83b027fd9c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:12:58.714870 14580 tablet_bootstrap.cc:492] T f9a010986c824dc58eb9a83b027fd9c2 P 4037cbe8f1de462ca703b3580eeb1af5: Bootstrap starting.
I20260925 15:12:58.716143 14580 tablet_bootstrap.cc:654] T f9a010986c824dc58eb9a83b027fd9c2 P 4037cbe8f1de462ca703b3580eeb1af5: Neither blocks nor log segments found. Creating new log.
I20260925 15:12:58.716872 14581 tablet_bootstrap.cc:492] T f9a010986c824dc58eb9a83b027fd9c2 P 3e607a2588f04ef7bd2c71fb7752782d: Bootstrap starting.
I20260925 15:12:58.717127 14579 tablet_bootstrap.cc:492] T f9a010986c824dc58eb9a83b027fd9c2 P 60ceefa28bab42e0adcb2b6345c27a62: Bootstrap starting.
I20260925 15:12:58.718288 14579 tablet_bootstrap.cc:654] T f9a010986c824dc58eb9a83b027fd9c2 P 60ceefa28bab42e0adcb2b6345c27a62: Neither blocks nor log segments found. Creating new log.
I20260925 15:12:58.718897 14581 tablet_bootstrap.cc:654] T f9a010986c824dc58eb9a83b027fd9c2 P 3e607a2588f04ef7bd2c71fb7752782d: Neither blocks nor log segments found. Creating new log.
I20260925 15:12:58.724454 14579 tablet_bootstrap.cc:492] T f9a010986c824dc58eb9a83b027fd9c2 P 60ceefa28bab42e0adcb2b6345c27a62: No bootstrap required, opened a new log
I20260925 15:12:58.724625 14579 ts_tablet_manager.cc:1397] T f9a010986c824dc58eb9a83b027fd9c2 P 60ceefa28bab42e0adcb2b6345c27a62: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.005s
I20260925 15:12:58.725057 14581 tablet_bootstrap.cc:492] T f9a010986c824dc58eb9a83b027fd9c2 P 3e607a2588f04ef7bd2c71fb7752782d: No bootstrap required, opened a new log
I20260925 15:12:58.725217 14581 ts_tablet_manager.cc:1397] T f9a010986c824dc58eb9a83b027fd9c2 P 3e607a2588f04ef7bd2c71fb7752782d: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.003s
I20260925 15:12:58.725437 14580 tablet_bootstrap.cc:492] T f9a010986c824dc58eb9a83b027fd9c2 P 4037cbe8f1de462ca703b3580eeb1af5: No bootstrap required, opened a new log
I20260925 15:12:58.725476 14579 raft_consensus.cc:348] T f9a010986c824dc58eb9a83b027fd9c2 P 60ceefa28bab42e0adcb2b6345c27a62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ceefa28bab42e0adcb2b6345c27a62" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 34225 } } peers { permanent_uuid: "4037cbe8f1de462ca703b3580eeb1af5" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 43891 } } peers { permanent_uuid: "3e607a2588f04ef7bd2c71fb7752782d" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43695 } }
I20260925 15:12:58.725719 14579 raft_consensus.cc:374] T f9a010986c824dc58eb9a83b027fd9c2 P 60ceefa28bab42e0adcb2b6345c27a62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:12:58.725816 14580 ts_tablet_manager.cc:1397] T f9a010986c824dc58eb9a83b027fd9c2 P 4037cbe8f1de462ca703b3580eeb1af5: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.005s
I20260925 15:12:58.725812 14579 raft_consensus.cc:729] T f9a010986c824dc58eb9a83b027fd9c2 P 60ceefa28bab42e0adcb2b6345c27a62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ceefa28bab42e0adcb2b6345c27a62, State: Initialized, Role: FOLLOWER
I20260925 15:12:58.726020 14581 raft_consensus.cc:348] T f9a010986c824dc58eb9a83b027fd9c2 P 3e607a2588f04ef7bd2c71fb7752782d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ceefa28bab42e0adcb2b6345c27a62" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 34225 } } peers { permanent_uuid: "4037cbe8f1de462ca703b3580eeb1af5" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 43891 } } peers { permanent_uuid: "3e607a2588f04ef7bd2c71fb7752782d" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43695 } }
I20260925 15:12:58.726107 14579 consensus_queue.cc:260] T f9a010986c824dc58eb9a83b027fd9c2 P 60ceefa28bab42e0adcb2b6345c27a62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ceefa28bab42e0adcb2b6345c27a62" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 34225 } } peers { permanent_uuid: "4037cbe8f1de462ca703b3580eeb1af5" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 43891 } } peers { permanent_uuid: "3e607a2588f04ef7bd2c71fb7752782d" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43695 } }
I20260925 15:12:58.726281 14581 raft_consensus.cc:374] T f9a010986c824dc58eb9a83b027fd9c2 P 3e607a2588f04ef7bd2c71fb7752782d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:12:58.726383 14581 raft_consensus.cc:729] T f9a010986c824dc58eb9a83b027fd9c2 P 3e607a2588f04ef7bd2c71fb7752782d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e607a2588f04ef7bd2c71fb7752782d, State: Initialized, Role: FOLLOWER
I20260925 15:12:58.726593 14581 consensus_queue.cc:260] T f9a010986c824dc58eb9a83b027fd9c2 P 3e607a2588f04ef7bd2c71fb7752782d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ceefa28bab42e0adcb2b6345c27a62" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 34225 } } peers { permanent_uuid: "4037cbe8f1de462ca703b3580eeb1af5" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 43891 } } peers { permanent_uuid: "3e607a2588f04ef7bd2c71fb7752782d" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43695 } }
I20260925 15:12:58.726636 14580 raft_consensus.cc:348] T f9a010986c824dc58eb9a83b027fd9c2 P 4037cbe8f1de462ca703b3580eeb1af5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ceefa28bab42e0adcb2b6345c27a62" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 34225 } } peers { permanent_uuid: "4037cbe8f1de462ca703b3580eeb1af5" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 43891 } } peers { permanent_uuid: "3e607a2588f04ef7bd2c71fb7752782d" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43695 } }
I20260925 15:12:58.726843 14580 raft_consensus.cc:374] T f9a010986c824dc58eb9a83b027fd9c2 P 4037cbe8f1de462ca703b3580eeb1af5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:12:58.726935 14580 raft_consensus.cc:729] T f9a010986c824dc58eb9a83b027fd9c2 P 4037cbe8f1de462ca703b3580eeb1af5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4037cbe8f1de462ca703b3580eeb1af5, State: Initialized, Role: FOLLOWER
I20260925 15:12:58.727135 14580 consensus_queue.cc:260] T f9a010986c824dc58eb9a83b027fd9c2 P 4037cbe8f1de462ca703b3580eeb1af5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ceefa28bab42e0adcb2b6345c27a62" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 34225 } } peers { permanent_uuid: "4037cbe8f1de462ca703b3580eeb1af5" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 43891 } } peers { permanent_uuid: "3e607a2588f04ef7bd2c71fb7752782d" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43695 } }
I20260925 15:12:58.727068 14586 tablet_replica.cc:439] T f9a010986c824dc58eb9a83b027fd9c2 P 3e607a2588f04ef7bd2c71fb7752782d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ceefa28bab42e0adcb2b6345c27a62" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 34225 } } peers { permanent_uuid: "4037cbe8f1de462ca703b3580eeb1af5" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 43891 } } peers { permanent_uuid: "3e607a2588f04ef7bd2c71fb7752782d" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43695 } } }
I20260925 15:12:58.727424 14586 tablet_replica.cc:442] T f9a010986c824dc58eb9a83b027fd9c2 P 3e607a2588f04ef7bd2c71fb7752782d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:12:58.727717 14580 ts_tablet_manager.cc:1428] T f9a010986c824dc58eb9a83b027fd9c2 P 4037cbe8f1de462ca703b3580eeb1af5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260925 15:12:58.727712 14587 tablet_replica.cc:439] T f9a010986c824dc58eb9a83b027fd9c2 P 4037cbe8f1de462ca703b3580eeb1af5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ceefa28bab42e0adcb2b6345c27a62" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 34225 } } peers { permanent_uuid: "4037cbe8f1de462ca703b3580eeb1af5" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 43891 } } peers { permanent_uuid: "3e607a2588f04ef7bd2c71fb7752782d" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43695 } } }
I20260925 15:12:58.728008 14587 tablet_replica.cc:442] T f9a010986c824dc58eb9a83b027fd9c2 P 4037cbe8f1de462ca703b3580eeb1af5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:12:58.728322 14581 ts_tablet_manager.cc:1428] T f9a010986c824dc58eb9a83b027fd9c2 P 3e607a2588f04ef7bd2c71fb7752782d: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 15:12:58.726564 14585 tablet_replica.cc:439] T f9a010986c824dc58eb9a83b027fd9c2 P 60ceefa28bab42e0adcb2b6345c27a62: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ceefa28bab42e0adcb2b6345c27a62" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 34225 } } peers { permanent_uuid: "4037cbe8f1de462ca703b3580eeb1af5" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 43891 } } peers { permanent_uuid: "3e607a2588f04ef7bd2c71fb7752782d" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43695 } } }
I20260925 15:12:58.728884 14585 tablet_replica.cc:442] T f9a010986c824dc58eb9a83b027fd9c2 P 60ceefa28bab42e0adcb2b6345c27a62: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:12:58.729417 14579 ts_tablet_manager.cc:1428] T f9a010986c824dc58eb9a83b027fd9c2 P 60ceefa28bab42e0adcb2b6345c27a62: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260925 15:12:58.795559 14585 raft_consensus.cc:482] T f9a010986c824dc58eb9a83b027fd9c2 P 60ceefa28bab42e0adcb2b6345c27a62 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:12:58.795836 14585 raft_consensus.cc:504] T f9a010986c824dc58eb9a83b027fd9c2 P 60ceefa28bab42e0adcb2b6345c27a62 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ceefa28bab42e0adcb2b6345c27a62" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 34225 } } peers { permanent_uuid: "4037cbe8f1de462ca703b3580eeb1af5" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 43891 } } peers { permanent_uuid: "3e607a2588f04ef7bd2c71fb7752782d" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43695 } }
I20260925 15:12:58.797276 14585 leader_election.cc:288] T f9a010986c824dc58eb9a83b027fd9c2 P 60ceefa28bab42e0adcb2b6345c27a62 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4037cbe8f1de462ca703b3580eeb1af5 (127.12.8.2:43891), 3e607a2588f04ef7bd2c71fb7752782d (127.12.8.3:43695)
I20260925 15:12:58.803962 14577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:58.797672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.1:51405 (local address 127.12.8.2:43891)
0925 15:12:58.797881 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:58.797888 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:58.798579 (+ 691us) server_negotiation.cc:407] Connection header received
0925 15:12:58.798721 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:58.798725 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:58.798812 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:58.798909 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:12:58.799607 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:58.800318 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:58.802104 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:58.802989 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:58.803143 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:58.803247 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:58.803567 (+ 320us) server_negotiation.cc:299] Negotiation successful
0925 15:12:58.803700 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260925 15:12:58.803984 14578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:58.797702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.1:49621 (local address 127.12.8.3:43695)
0925 15:12:58.797882 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:58.797889 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:58.798425 (+ 536us) server_negotiation.cc:407] Connection header received
0925 15:12:58.798587 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:58.798593 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:58.798686 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:58.798773 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:12:58.799418 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:58.800173 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:58.802102 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:58.803091 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:58.803234 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:58.803339 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:58.803604 (+ 265us) server_negotiation.cc:299] Negotiation successful
0925 15:12:58.803723 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 15:12:58.804073 14602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:58.797507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.2:43891 (local address 127.12.8.1:51405)
0925 15:12:58.798437 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:58.798456 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 15:12:58.798603 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:12:58.799008 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:12:58.799013 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:12:58.799041 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:12:58.799447 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:58.799456 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:58.800461 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:58.800466 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:58.801954 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:58.801966 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:58.803203 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:58.803209 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:58.803419 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:58.803442 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 15:12:58.803544 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 15:12:58.803663 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":762,"mutex_wait_us":90,"thread_start_us":542,"threads_started":1}
I20260925 15:12:58.804332 14601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:58.797501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.3:43695 (local address 127.12.8.1:49621)
0925 15:12:58.798323 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:58.798340 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 15:12:58.798455 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:12:58.798799 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:12:58.798803 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:12:58.798845 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:12:58.799256 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:58.799268 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:58.800296 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:58.800304 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:58.801961 (+ 1657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:58.801972 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:58.803317 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:58.803323 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:58.803459 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:58.803480 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 15:12:58.803582 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 15:12:58.803700 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"mutex_wait_us":108,"thread_start_us":440,"threads_started":1}
I20260925 15:12:58.804936 14515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9a010986c824dc58eb9a83b027fd9c2" candidate_uuid: "60ceefa28bab42e0adcb2b6345c27a62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e607a2588f04ef7bd2c71fb7752782d" is_pre_election: true
I20260925 15:12:58.804947 14376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9a010986c824dc58eb9a83b027fd9c2" candidate_uuid: "60ceefa28bab42e0adcb2b6345c27a62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4037cbe8f1de462ca703b3580eeb1af5" is_pre_election: true
I20260925 15:12:58.805207 14515 raft_consensus.cc:2454] T f9a010986c824dc58eb9a83b027fd9c2 P 3e607a2588f04ef7bd2c71fb7752782d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60ceefa28bab42e0adcb2b6345c27a62 in term 0.
I20260925 15:12:58.805358 14376 raft_consensus.cc:2454] T f9a010986c824dc58eb9a83b027fd9c2 P 4037cbe8f1de462ca703b3580eeb1af5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60ceefa28bab42e0adcb2b6345c27a62 in term 0.
I20260925 15:12:58.805651 14163 leader_election.cc:302] T f9a010986c824dc58eb9a83b027fd9c2 P 60ceefa28bab42e0adcb2b6345c27a62 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e607a2588f04ef7bd2c71fb7752782d, 60ceefa28bab42e0adcb2b6345c27a62; no voters:
I20260925 15:12:58.806022 14585 raft_consensus.cc:2781] T f9a010986c824dc58eb9a83b027fd9c2 P 60ceefa28bab42e0adcb2b6345c27a62 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:12:58.806119 14585 raft_consensus.cc:482] T f9a010986c824dc58eb9a83b027fd9c2 P 60ceefa28bab42e0adcb2b6345c27a62 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:12:58.806250 14585 raft_consensus.cc:3037] T f9a010986c824dc58eb9a83b027fd9c2 P 60ceefa28bab42e0adcb2b6345c27a62 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:12:58.808250 14585 raft_consensus.cc:504] T f9a010986c824dc58eb9a83b027fd9c2 P 60ceefa28bab42e0adcb2b6345c27a62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ceefa28bab42e0adcb2b6345c27a62" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 34225 } } peers { permanent_uuid: "4037cbe8f1de462ca703b3580eeb1af5" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 43891 } } peers { permanent_uuid: "3e607a2588f04ef7bd2c71fb7752782d" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43695 } }
I20260925 15:12:58.808790 14585 leader_election.cc:288] T f9a010986c824dc58eb9a83b027fd9c2 P 60ceefa28bab42e0adcb2b6345c27a62 [CANDIDATE]: Term 1 election: Requested vote from peers 4037cbe8f1de462ca703b3580eeb1af5 (127.12.8.2:43891), 3e607a2588f04ef7bd2c71fb7752782d (127.12.8.3:43695)
I20260925 15:12:58.809247 14376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9a010986c824dc58eb9a83b027fd9c2" candidate_uuid: "60ceefa28bab42e0adcb2b6345c27a62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4037cbe8f1de462ca703b3580eeb1af5"
I20260925 15:12:58.809231 14515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9a010986c824dc58eb9a83b027fd9c2" candidate_uuid: "60ceefa28bab42e0adcb2b6345c27a62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e607a2588f04ef7bd2c71fb7752782d"
I20260925 15:12:58.809423 14376 raft_consensus.cc:3037] T f9a010986c824dc58eb9a83b027fd9c2 P 4037cbe8f1de462ca703b3580eeb1af5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:12:58.809441 14515 raft_consensus.cc:3037] T f9a010986c824dc58eb9a83b027fd9c2 P 3e607a2588f04ef7bd2c71fb7752782d [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:12:58.812522 14515 raft_consensus.cc:2454] T f9a010986c824dc58eb9a83b027fd9c2 P 3e607a2588f04ef7bd2c71fb7752782d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60ceefa28bab42e0adcb2b6345c27a62 in term 1.
I20260925 15:12:58.812536 14376 raft_consensus.cc:2454] T f9a010986c824dc58eb9a83b027fd9c2 P 4037cbe8f1de462ca703b3580eeb1af5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60ceefa28bab42e0adcb2b6345c27a62 in term 1.
I20260925 15:12:58.812903 14163 leader_election.cc:302] T f9a010986c824dc58eb9a83b027fd9c2 P 60ceefa28bab42e0adcb2b6345c27a62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e607a2588f04ef7bd2c71fb7752782d, 60ceefa28bab42e0adcb2b6345c27a62; no voters:
I20260925 15:12:58.813202 14585 raft_consensus.cc:2781] T f9a010986c824dc58eb9a83b027fd9c2 P 60ceefa28bab42e0adcb2b6345c27a62 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:12:58.813716 14585 raft_consensus.cc:686] T f9a010986c824dc58eb9a83b027fd9c2 P 60ceefa28bab42e0adcb2b6345c27a62 [term 1 LEADER]: Becoming Leader. State: Replica: 60ceefa28bab42e0adcb2b6345c27a62, State: Running, Role: LEADER
I20260925 15:12:58.814260 14585 consensus_queue.cc:237] T f9a010986c824dc58eb9a83b027fd9c2 P 60ceefa28bab42e0adcb2b6345c27a62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ceefa28bab42e0adcb2b6345c27a62" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 34225 } } peers { permanent_uuid: "4037cbe8f1de462ca703b3580eeb1af5" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 43891 } } peers { permanent_uuid: "3e607a2588f04ef7bd2c71fb7752782d" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43695 } }
I20260925 15:12:58.817070 14603 tablet_replica.cc:439] T f9a010986c824dc58eb9a83b027fd9c2 P 60ceefa28bab42e0adcb2b6345c27a62: TxnStatusTablet state changed. Reason: New leader 60ceefa28bab42e0adcb2b6345c27a62. Latest consensus state: current_term: 1 leader_uuid: "60ceefa28bab42e0adcb2b6345c27a62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ceefa28bab42e0adcb2b6345c27a62" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 34225 } } peers { permanent_uuid: "4037cbe8f1de462ca703b3580eeb1af5" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 43891 } } peers { permanent_uuid: "3e607a2588f04ef7bd2c71fb7752782d" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43695 } } }
I20260925 15:12:58.817437 14603 tablet_replica.cc:442] T f9a010986c824dc58eb9a83b027fd9c2 P 60ceefa28bab42e0adcb2b6345c27a62: This TxnStatusTablet replica's current role is: LEADER
I20260925 15:12:58.818222 14605 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 15:12:58.818492 14605 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 15:12:58.819023 14073 catalog_manager.cc:5495] T f9a010986c824dc58eb9a83b027fd9c2 P 60ceefa28bab42e0adcb2b6345c27a62 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60ceefa28bab42e0adcb2b6345c27a62 (127.12.8.1). New cstate: current_term: 1 leader_uuid: "60ceefa28bab42e0adcb2b6345c27a62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ceefa28bab42e0adcb2b6345c27a62" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 34225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4037cbe8f1de462ca703b3580eeb1af5" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 43891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e607a2588f04ef7bd2c71fb7752782d" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43695 } health_report { overall_health: UNKNOWN } } }
I20260925 15:12:58.852164 14597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:58.846210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.1:34225 (local address 127.0.0.1:48966)
0925 15:12:58.846453 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:58.846473 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 15:12:58.846584 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:12:58.847076 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:12:58.847080 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:12:58.847090 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 15:12:58.847236 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:58.847243 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:58.848661 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:58.848665 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:58.849636 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:58.849647 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:58.850602 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:58.850606 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:58.850759 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:58.850808 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 15:12:58.851774 (+ 966us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 15:12:58.851788 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 15:12:58.851842 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 15:12:58.851945 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260925 15:12:58.852285 14574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:58.846399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48966 (local address 127.12.8.1:34225)
0925 15:12:58.846566 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:58.846572 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:58.846593 (+ 21us) server_negotiation.cc:407] Connection header received
0925 15:12:58.846768 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:58.846774 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:58.846903 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:58.847053 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:12:58.847358 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:58.848525 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:58.849772 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:58.850460 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:58.850585 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:58.850988 (+ 403us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:12:58.851666 (+ 678us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:12:58.851817 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:58.851865 (+ 48us) server_negotiation.cc:299] Negotiation successful
0925 15:12:58.851950 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 15:12:58.859891 14376 raft_consensus.cc:1261] T f9a010986c824dc58eb9a83b027fd9c2 P 4037cbe8f1de462ca703b3580eeb1af5 [term 1 FOLLOWER]: Refusing update from remote peer 60ceefa28bab42e0adcb2b6345c27a62: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:12:58.860167 14515 raft_consensus.cc:1261] T f9a010986c824dc58eb9a83b027fd9c2 P 3e607a2588f04ef7bd2c71fb7752782d [term 1 FOLLOWER]: Refusing update from remote peer 60ceefa28bab42e0adcb2b6345c27a62: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:12:58.860873 14603 consensus_queue.cc:1035] T f9a010986c824dc58eb9a83b027fd9c2 P 60ceefa28bab42e0adcb2b6345c27a62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4037cbe8f1de462ca703b3580eeb1af5" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 43891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:12:58.861200 14585 consensus_queue.cc:1035] T f9a010986c824dc58eb9a83b027fd9c2 P 60ceefa28bab42e0adcb2b6345c27a62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e607a2588f04ef7bd2c71fb7752782d" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:12:58.871152 14586 tablet_replica.cc:439] T f9a010986c824dc58eb9a83b027fd9c2 P 3e607a2588f04ef7bd2c71fb7752782d: TxnStatusTablet state changed. Reason: New leader 60ceefa28bab42e0adcb2b6345c27a62. Latest consensus state: current_term: 1 leader_uuid: "60ceefa28bab42e0adcb2b6345c27a62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ceefa28bab42e0adcb2b6345c27a62" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 34225 } } peers { permanent_uuid: "4037cbe8f1de462ca703b3580eeb1af5" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 43891 } } peers { permanent_uuid: "3e607a2588f04ef7bd2c71fb7752782d" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43695 } } }
I20260925 15:12:58.871537 14586 tablet_replica.cc:442] T f9a010986c824dc58eb9a83b027fd9c2 P 3e607a2588f04ef7bd2c71fb7752782d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:12:58.872991 14587 tablet_replica.cc:439] T f9a010986c824dc58eb9a83b027fd9c2 P 4037cbe8f1de462ca703b3580eeb1af5: TxnStatusTablet state changed. Reason: New leader 60ceefa28bab42e0adcb2b6345c27a62. Latest consensus state: current_term: 1 leader_uuid: "60ceefa28bab42e0adcb2b6345c27a62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ceefa28bab42e0adcb2b6345c27a62" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 34225 } } peers { permanent_uuid: "4037cbe8f1de462ca703b3580eeb1af5" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 43891 } } peers { permanent_uuid: "3e607a2588f04ef7bd2c71fb7752782d" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43695 } } }
I20260925 15:12:58.873283 14587 tablet_replica.cc:442] T f9a010986c824dc58eb9a83b027fd9c2 P 4037cbe8f1de462ca703b3580eeb1af5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:12:58.883733 14585 tablet_replica.cc:439] T f9a010986c824dc58eb9a83b027fd9c2 P 60ceefa28bab42e0adcb2b6345c27a62: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "60ceefa28bab42e0adcb2b6345c27a62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ceefa28bab42e0adcb2b6345c27a62" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 34225 } } peers { permanent_uuid: "4037cbe8f1de462ca703b3580eeb1af5" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 43891 } } peers { permanent_uuid: "3e607a2588f04ef7bd2c71fb7752782d" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43695 } } }
I20260925 15:12:58.884223 14585 tablet_replica.cc:442] T f9a010986c824dc58eb9a83b027fd9c2 P 60ceefa28bab42e0adcb2b6345c27a62: This TxnStatusTablet replica's current role is: LEADER
I20260925 15:12:58.884774 14603 tablet_replica.cc:439] T f9a010986c824dc58eb9a83b027fd9c2 P 60ceefa28bab42e0adcb2b6345c27a62: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "60ceefa28bab42e0adcb2b6345c27a62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ceefa28bab42e0adcb2b6345c27a62" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 34225 } } peers { permanent_uuid: "4037cbe8f1de462ca703b3580eeb1af5" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 43891 } } peers { permanent_uuid: "3e607a2588f04ef7bd2c71fb7752782d" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43695 } } }
I20260925 15:12:58.885103 14603 tablet_replica.cc:442] T f9a010986c824dc58eb9a83b027fd9c2 P 60ceefa28bab42e0adcb2b6345c27a62: This TxnStatusTablet replica's current role is: LEADER
I20260925 15:12:58.893034 14586 tablet_replica.cc:439] T f9a010986c824dc58eb9a83b027fd9c2 P 3e607a2588f04ef7bd2c71fb7752782d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "60ceefa28bab42e0adcb2b6345c27a62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ceefa28bab42e0adcb2b6345c27a62" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 34225 } } peers { permanent_uuid: "4037cbe8f1de462ca703b3580eeb1af5" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 43891 } } peers { permanent_uuid: "3e607a2588f04ef7bd2c71fb7752782d" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43695 } } }
I20260925 15:12:58.893281 14586 tablet_replica.cc:442] T f9a010986c824dc58eb9a83b027fd9c2 P 3e607a2588f04ef7bd2c71fb7752782d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:12:58.897608 14587 tablet_replica.cc:439] T f9a010986c824dc58eb9a83b027fd9c2 P 4037cbe8f1de462ca703b3580eeb1af5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "60ceefa28bab42e0adcb2b6345c27a62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ceefa28bab42e0adcb2b6345c27a62" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 34225 } } peers { permanent_uuid: "4037cbe8f1de462ca703b3580eeb1af5" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 43891 } } peers { permanent_uuid: "3e607a2588f04ef7bd2c71fb7752782d" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43695 } } }
I20260925 15:12:58.897886 14587 tablet_replica.cc:442] T f9a010986c824dc58eb9a83b027fd9c2 P 4037cbe8f1de462ca703b3580eeb1af5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:12:58.901130 14577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:58.893653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49176 (local address 127.12.8.2:43891)
0925 15:12:58.893867 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:58.893876 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:58.893896 (+ 20us) server_negotiation.cc:407] Connection header received
0925 15:12:58.893983 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:58.893988 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:58.894119 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:58.894301 (+ 182us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:12:58.895057 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:58.896173 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:58.898343 (+ 2170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:58.898948 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:58.899095 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:58.899724 (+ 629us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:12:58.900544 (+ 820us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:12:58.900692 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:58.900771 (+ 79us) server_negotiation.cc:299] Negotiation successful
0925 15:12:58.900855 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260925 15:12:58.916623 14574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:58.911397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.2:60621 (local address 127.12.8.1:34225)
0925 15:12:58.911599 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:58.911606 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:58.911922 (+ 316us) server_negotiation.cc:407] Connection header received
0925 15:12:58.912147 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:58.912151 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:58.912244 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:58.912336 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:12:58.912817 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:58.913570 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:58.914953 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:58.915376 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:58.915514 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:58.915922 (+ 408us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:12:58.916020 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:12:58.916115 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:58.916244 (+ 129us) server_negotiation.cc:299] Negotiation successful
0925 15:12:58.916319 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 15:12:58.916721 14620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:58.911263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.1:34225 (local address 127.12.8.2:60621)
0925 15:12:58.911821 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:58.911842 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 15:12:58.911992 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:12:58.912356 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:12:58.912360 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:12:58.912372 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 15:12:58.912681 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:58.912689 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:58.913693 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:58.913702 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:58.914807 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:58.914817 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:58.915543 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:58.915550 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:58.915685 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:58.915730 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 15:12:58.916115 (+ 385us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 15:12:58.916128 (+ 13us) client_negotiation.cc:769] Sending connection context
0925 15:12:58.916220 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 15:12:58.916352 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":268,"threads_started":1}
I20260925 15:12:58.926745 14515 raft_consensus.cc:1261] T 10867a35e08f46f0801bd1fc11103f2c P 3e607a2588f04ef7bd2c71fb7752782d [term 1 FOLLOWER]: Refusing update from remote peer 4037cbe8f1de462ca703b3580eeb1af5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:12:58.927043 14232 raft_consensus.cc:1261] T 10867a35e08f46f0801bd1fc11103f2c P 60ceefa28bab42e0adcb2b6345c27a62 [term 1 FOLLOWER]: Refusing update from remote peer 4037cbe8f1de462ca703b3580eeb1af5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:12:58.927469 14590 consensus_queue.cc:1035] T 10867a35e08f46f0801bd1fc11103f2c P 4037cbe8f1de462ca703b3580eeb1af5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e607a2588f04ef7bd2c71fb7752782d" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:12:58.927914 14587 consensus_queue.cc:1035] T 10867a35e08f46f0801bd1fc11103f2c P 4037cbe8f1de462ca703b3580eeb1af5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ceefa28bab42e0adcb2b6345c27a62" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 34225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:12:58.934821 14609 mvcc.cc:205] Tried to move back new op lower bound from 7333270236876955648 to 7333270235720404992. Current Snapshot: MvccSnapshot[applied={T|T < 7333270236876955648}]
I20260925 15:12:58.975919 14577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:58.967740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.1:37019 (local address 127.12.8.2:43891)
0925 15:12:58.967903 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:58.967909 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:58.968996 (+ 1087us) server_negotiation.cc:407] Connection header received
0925 15:12:58.969256 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:58.969262 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:58.969334 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:58.969387 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:12:58.970054 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:58.971238 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:58.973692 (+ 2454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:58.974413 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:58.974498 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:58.975157 (+ 659us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:12:58.975289 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:12:58.975346 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:58.975541 (+ 195us) server_negotiation.cc:299] Negotiation successful
0925 15:12:58.975640 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 15:12:58.977375 14625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:58.967824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.2:43891 (local address 127.12.8.1:37019)
0925 15:12:58.968909 (+ 1085us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:58.968928 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 15:12:58.969105 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:12:58.969552 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:12:58.969556 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:12:58.969568 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 15:12:58.969893 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:58.969903 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:58.971434 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:58.971441 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:58.973538 (+ 2097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:58.973555 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:58.974707 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:58.974713 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:58.974873 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:58.974937 (+ 64us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 15:12:58.975424 (+ 487us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 15:12:58.975433 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 15:12:58.976742 (+ 1309us) client_negotiation.cc:240] Negotiation successful
0925 15:12:58.976931 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":904,"thread_start_us":741,"threads_started":1}
I20260925 15:12:59.021153 14075 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42528:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:12:59.021492 14075 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42528:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:12:59.025584 14075 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8ee449b6924246da81d3c6ac299728f6: Sending DeleteTablet for 3 replicas of tablet 10867a35e08f46f0801bd1fc11103f2c
I20260925 15:12:59.026778 14355 tablet_service.cc:1514] Processing DeleteTablet for tablet 10867a35e08f46f0801bd1fc11103f2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:12:59 UTC) from {username='kudu', principal='kudu/127.12.8.62@KRBTEST.COM'} at 127.0.0.1:49170
I20260925 15:12:59.026791 14490 tablet_service.cc:1514] Processing DeleteTablet for tablet 10867a35e08f46f0801bd1fc11103f2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:12:59 UTC) from {username='kudu', principal='kudu/127.12.8.62@KRBTEST.COM'} at 127.0.0.1:60456
I20260925 15:12:59.027856 14212 tablet_service.cc:1514] Processing DeleteTablet for tablet 10867a35e08f46f0801bd1fc11103f2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:12:59 UTC) from {username='kudu', principal='kudu/127.12.8.62@KRBTEST.COM'} at 127.0.0.1:48958
I20260925 15:12:59.035912 14075 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42528:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:12:59.038077 14075 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:12:59.049898 14489 tablet_service.cc:1467] Processing CreateTablet for tablet 57d7907070e14f2c9fb6528cb7355784 (DEFAULT_TABLE table=test-table [id=83eca47c4bc44f6ebfede9ec9c3b0bc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:12:59.049961 14350 tablet_service.cc:1467] Processing CreateTablet for tablet 57d7907070e14f2c9fb6528cb7355784 (DEFAULT_TABLE table=test-table [id=83eca47c4bc44f6ebfede9ec9c3b0bc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:12:59.050522 14489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57d7907070e14f2c9fb6528cb7355784. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:12:59.050577 14350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57d7907070e14f2c9fb6528cb7355784. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:12:59.054979 14211 tablet_service.cc:1467] Processing CreateTablet for tablet 57d7907070e14f2c9fb6528cb7355784 (DEFAULT_TABLE table=test-table [id=83eca47c4bc44f6ebfede9ec9c3b0bc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:12:59.055562 14211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57d7907070e14f2c9fb6528cb7355784. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:12:59.056141 14630 tablet_replica.cc:330] T 10867a35e08f46f0801bd1fc11103f2c P 60ceefa28bab42e0adcb2b6345c27a62: stopping tablet replica
I20260925 15:12:59.056511 14629 tablet_replica.cc:330] T 10867a35e08f46f0801bd1fc11103f2c P 3e607a2588f04ef7bd2c71fb7752782d: stopping tablet replica
I20260925 15:12:59.057008 14629 raft_consensus.cc:2229] T 10867a35e08f46f0801bd1fc11103f2c P 3e607a2588f04ef7bd2c71fb7752782d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:12:59.057528 14630 raft_consensus.cc:2229] T 10867a35e08f46f0801bd1fc11103f2c P 60ceefa28bab42e0adcb2b6345c27a62 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:12:59.057988 14629 raft_consensus.cc:2258] T 10867a35e08f46f0801bd1fc11103f2c P 3e607a2588f04ef7bd2c71fb7752782d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:12:59.058341 14630 raft_consensus.cc:2258] T 10867a35e08f46f0801bd1fc11103f2c P 60ceefa28bab42e0adcb2b6345c27a62 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:12:59.059751 14581 tablet_bootstrap.cc:492] T 57d7907070e14f2c9fb6528cb7355784 P 3e607a2588f04ef7bd2c71fb7752782d: Bootstrap starting.
I20260925 15:12:59.061595 14581 tablet_bootstrap.cc:654] T 57d7907070e14f2c9fb6528cb7355784 P 3e607a2588f04ef7bd2c71fb7752782d: Neither blocks nor log segments found. Creating new log.
I20260925 15:12:59.062424 14630 ts_tablet_manager.cc:1905] T 10867a35e08f46f0801bd1fc11103f2c P 60ceefa28bab42e0adcb2b6345c27a62: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:12:59.062716 14629 ts_tablet_manager.cc:1905] T 10867a35e08f46f0801bd1fc11103f2c P 3e607a2588f04ef7bd2c71fb7752782d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:12:59.066244 14579 tablet_bootstrap.cc:492] T 57d7907070e14f2c9fb6528cb7355784 P 60ceefa28bab42e0adcb2b6345c27a62: Bootstrap starting.
I20260925 15:12:59.066751 14628 tablet_replica.cc:330] T 10867a35e08f46f0801bd1fc11103f2c P 4037cbe8f1de462ca703b3580eeb1af5: stopping tablet replica
I20260925 15:12:59.068024 14579 tablet_bootstrap.cc:654] T 57d7907070e14f2c9fb6528cb7355784 P 60ceefa28bab42e0adcb2b6345c27a62: Neither blocks nor log segments found. Creating new log.
I20260925 15:12:59.068502 14628 raft_consensus.cc:2229] T 10867a35e08f46f0801bd1fc11103f2c P 4037cbe8f1de462ca703b3580eeb1af5 [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:12:59.069231 14628 raft_consensus.cc:2258] T 10867a35e08f46f0801bd1fc11103f2c P 4037cbe8f1de462ca703b3580eeb1af5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:12:59.071960 14580 tablet_bootstrap.cc:492] T 57d7907070e14f2c9fb6528cb7355784 P 4037cbe8f1de462ca703b3580eeb1af5: Bootstrap starting.
I20260925 15:12:59.075120 14581 tablet_bootstrap.cc:492] T 57d7907070e14f2c9fb6528cb7355784 P 3e607a2588f04ef7bd2c71fb7752782d: No bootstrap required, opened a new log
I20260925 15:12:59.075281 14581 ts_tablet_manager.cc:1397] T 57d7907070e14f2c9fb6528cb7355784 P 3e607a2588f04ef7bd2c71fb7752782d: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.004s
I20260925 15:12:59.076000 14580 tablet_bootstrap.cc:654] T 57d7907070e14f2c9fb6528cb7355784 P 4037cbe8f1de462ca703b3580eeb1af5: Neither blocks nor log segments found. Creating new log.
I20260925 15:12:59.076084 14581 raft_consensus.cc:348] T 57d7907070e14f2c9fb6528cb7355784 P 3e607a2588f04ef7bd2c71fb7752782d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ceefa28bab42e0adcb2b6345c27a62" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 34225 } } peers { permanent_uuid: "4037cbe8f1de462ca703b3580eeb1af5" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 43891 } } peers { permanent_uuid: "3e607a2588f04ef7bd2c71fb7752782d" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43695 } }
I20260925 15:12:59.076359 14581 raft_consensus.cc:374] T 57d7907070e14f2c9fb6528cb7355784 P 3e607a2588f04ef7bd2c71fb7752782d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:12:59.076431 14581 raft_consensus.cc:729] T 57d7907070e14f2c9fb6528cb7355784 P 3e607a2588f04ef7bd2c71fb7752782d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e607a2588f04ef7bd2c71fb7752782d, State: Initialized, Role: FOLLOWER
I20260925 15:12:59.076707 14581 consensus_queue.cc:260] T 57d7907070e14f2c9fb6528cb7355784 P 3e607a2588f04ef7bd2c71fb7752782d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ceefa28bab42e0adcb2b6345c27a62" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 34225 } } peers { permanent_uuid: "4037cbe8f1de462ca703b3580eeb1af5" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 43891 } } peers { permanent_uuid: "3e607a2588f04ef7bd2c71fb7752782d" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43695 } }
I20260925 15:12:59.079828 14581 ts_tablet_manager.cc:1428] T 57d7907070e14f2c9fb6528cb7355784 P 3e607a2588f04ef7bd2c71fb7752782d: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20260925 15:12:59.083694 14628 ts_tablet_manager.cc:1905] T 10867a35e08f46f0801bd1fc11103f2c P 4037cbe8f1de462ca703b3580eeb1af5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:12:59.084786 14579 tablet_bootstrap.cc:492] T 57d7907070e14f2c9fb6528cb7355784 P 60ceefa28bab42e0adcb2b6345c27a62: No bootstrap required, opened a new log
I20260925 15:12:59.084991 14579 ts_tablet_manager.cc:1397] T 57d7907070e14f2c9fb6528cb7355784 P 60ceefa28bab42e0adcb2b6345c27a62: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.005s
I20260925 15:12:59.085804 14579 raft_consensus.cc:348] T 57d7907070e14f2c9fb6528cb7355784 P 60ceefa28bab42e0adcb2b6345c27a62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ceefa28bab42e0adcb2b6345c27a62" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 34225 } } peers { permanent_uuid: "4037cbe8f1de462ca703b3580eeb1af5" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 43891 } } peers { permanent_uuid: "3e607a2588f04ef7bd2c71fb7752782d" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43695 } }
I20260925 15:12:59.086151 14579 raft_consensus.cc:374] T 57d7907070e14f2c9fb6528cb7355784 P 60ceefa28bab42e0adcb2b6345c27a62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:12:59.086269 14579 raft_consensus.cc:729] T 57d7907070e14f2c9fb6528cb7355784 P 60ceefa28bab42e0adcb2b6345c27a62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ceefa28bab42e0adcb2b6345c27a62, State: Initialized, Role: FOLLOWER
I20260925 15:12:59.086529 14579 consensus_queue.cc:260] T 57d7907070e14f2c9fb6528cb7355784 P 60ceefa28bab42e0adcb2b6345c27a62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ceefa28bab42e0adcb2b6345c27a62" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 34225 } } peers { permanent_uuid: "4037cbe8f1de462ca703b3580eeb1af5" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 43891 } } peers { permanent_uuid: "3e607a2588f04ef7bd2c71fb7752782d" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43695 } }
I20260925 15:12:59.087455 14580 tablet_bootstrap.cc:492] T 57d7907070e14f2c9fb6528cb7355784 P 4037cbe8f1de462ca703b3580eeb1af5: No bootstrap required, opened a new log
I20260925 15:12:59.087675 14580 ts_tablet_manager.cc:1397] T 57d7907070e14f2c9fb6528cb7355784 P 4037cbe8f1de462ca703b3580eeb1af5: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.005s
I20260925 15:12:59.088541 14580 raft_consensus.cc:348] T 57d7907070e14f2c9fb6528cb7355784 P 4037cbe8f1de462ca703b3580eeb1af5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ceefa28bab42e0adcb2b6345c27a62" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 34225 } } peers { permanent_uuid: "4037cbe8f1de462ca703b3580eeb1af5" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 43891 } } peers { permanent_uuid: "3e607a2588f04ef7bd2c71fb7752782d" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43695 } }
I20260925 15:12:59.088829 14580 raft_consensus.cc:374] T 57d7907070e14f2c9fb6528cb7355784 P 4037cbe8f1de462ca703b3580eeb1af5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:12:59.088933 14580 raft_consensus.cc:729] T 57d7907070e14f2c9fb6528cb7355784 P 4037cbe8f1de462ca703b3580eeb1af5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4037cbe8f1de462ca703b3580eeb1af5, State: Initialized, Role: FOLLOWER
I20260925 15:12:59.089174 14579 ts_tablet_manager.cc:1428] T 57d7907070e14f2c9fb6528cb7355784 P 60ceefa28bab42e0adcb2b6345c27a62: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260925 15:12:59.089246 14580 consensus_queue.cc:260] T 57d7907070e14f2c9fb6528cb7355784 P 4037cbe8f1de462ca703b3580eeb1af5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ceefa28bab42e0adcb2b6345c27a62" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 34225 } } peers { permanent_uuid: "4037cbe8f1de462ca703b3580eeb1af5" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 43891 } } peers { permanent_uuid: "3e607a2588f04ef7bd2c71fb7752782d" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43695 } }
I20260925 15:12:59.089986 14580 ts_tablet_manager.cc:1428] T 57d7907070e14f2c9fb6528cb7355784 P 4037cbe8f1de462ca703b3580eeb1af5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260925 15:12:59.092173 14630 ts_tablet_manager.cc:1918] T 10867a35e08f46f0801bd1fc11103f2c P 60ceefa28bab42e0adcb2b6345c27a62: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 15:12:59.092351 14630 log.cc:1196] T 10867a35e08f46f0801bd1fc11103f2c P 60ceefa28bab42e0adcb2b6345c27a62: Deleting WAL directory at /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/minicluster-data/ts-0/wal/wals/10867a35e08f46f0801bd1fc11103f2c
I20260925 15:12:59.092341 14629 ts_tablet_manager.cc:1918] T 10867a35e08f46f0801bd1fc11103f2c P 3e607a2588f04ef7bd2c71fb7752782d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 15:12:59.092653 14629 log.cc:1196] T 10867a35e08f46f0801bd1fc11103f2c P 3e607a2588f04ef7bd2c71fb7752782d: Deleting WAL directory at /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/minicluster-data/ts-2/wal/wals/10867a35e08f46f0801bd1fc11103f2c
I20260925 15:12:59.092922 14630 ts_tablet_manager.cc:1939] T 10867a35e08f46f0801bd1fc11103f2c P 60ceefa28bab42e0adcb2b6345c27a62: Deleting consensus metadata
I20260925 15:12:59.093307 14629 ts_tablet_manager.cc:1939] T 10867a35e08f46f0801bd1fc11103f2c P 3e607a2588f04ef7bd2c71fb7752782d: Deleting consensus metadata
I20260925 15:12:59.094676 14060 catalog_manager.cc:4841] TS 60ceefa28bab42e0adcb2b6345c27a62 (127.12.8.1:34225): tablet 10867a35e08f46f0801bd1fc11103f2c (table test-table [id=e0d8da9bc2eb4a6b89a0924652d8f8ea]) successfully deleted
I20260925 15:12:59.095376 14060 catalog_manager.cc:4841] TS 3e607a2588f04ef7bd2c71fb7752782d (127.12.8.3:43695): tablet 10867a35e08f46f0801bd1fc11103f2c (table test-table [id=e0d8da9bc2eb4a6b89a0924652d8f8ea]) successfully deleted
I20260925 15:12:59.101091 14628 ts_tablet_manager.cc:1918] T 10867a35e08f46f0801bd1fc11103f2c P 4037cbe8f1de462ca703b3580eeb1af5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 15:12:59.101267 14628 log.cc:1196] T 10867a35e08f46f0801bd1fc11103f2c P 4037cbe8f1de462ca703b3580eeb1af5: Deleting WAL directory at /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/minicluster-data/ts-1/wal/wals/10867a35e08f46f0801bd1fc11103f2c
I20260925 15:12:59.101867 14628 ts_tablet_manager.cc:1939] T 10867a35e08f46f0801bd1fc11103f2c P 4037cbe8f1de462ca703b3580eeb1af5: Deleting consensus metadata
I20260925 15:12:59.103456 14059 catalog_manager.cc:4841] TS 4037cbe8f1de462ca703b3580eeb1af5 (127.12.8.2:43891): tablet 10867a35e08f46f0801bd1fc11103f2c (table test-table [id=e0d8da9bc2eb4a6b89a0924652d8f8ea]) successfully deleted
I20260925 15:12:59.141284 14585 raft_consensus.cc:482] T 57d7907070e14f2c9fb6528cb7355784 P 60ceefa28bab42e0adcb2b6345c27a62 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:12:59.141506 14585 raft_consensus.cc:504] T 57d7907070e14f2c9fb6528cb7355784 P 60ceefa28bab42e0adcb2b6345c27a62 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ceefa28bab42e0adcb2b6345c27a62" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 34225 } } peers { permanent_uuid: "4037cbe8f1de462ca703b3580eeb1af5" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 43891 } } peers { permanent_uuid: "3e607a2588f04ef7bd2c71fb7752782d" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43695 } }
I20260925 15:12:59.142264 14585 leader_election.cc:288] T 57d7907070e14f2c9fb6528cb7355784 P 60ceefa28bab42e0adcb2b6345c27a62 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4037cbe8f1de462ca703b3580eeb1af5 (127.12.8.2:43891), 3e607a2588f04ef7bd2c71fb7752782d (127.12.8.3:43695)
I20260925 15:12:59.142759 14376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57d7907070e14f2c9fb6528cb7355784" candidate_uuid: "60ceefa28bab42e0adcb2b6345c27a62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4037cbe8f1de462ca703b3580eeb1af5" is_pre_election: true
I20260925 15:12:59.143006 14376 raft_consensus.cc:2454] T 57d7907070e14f2c9fb6528cb7355784 P 4037cbe8f1de462ca703b3580eeb1af5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60ceefa28bab42e0adcb2b6345c27a62 in term 0.
I20260925 15:12:59.142954 14514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57d7907070e14f2c9fb6528cb7355784" candidate_uuid: "60ceefa28bab42e0adcb2b6345c27a62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e607a2588f04ef7bd2c71fb7752782d" is_pre_election: true
I20260925 15:12:59.143198 14514 raft_consensus.cc:2454] T 57d7907070e14f2c9fb6528cb7355784 P 3e607a2588f04ef7bd2c71fb7752782d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60ceefa28bab42e0adcb2b6345c27a62 in term 0.
I20260925 15:12:59.143496 14162 leader_election.cc:302] T 57d7907070e14f2c9fb6528cb7355784 P 60ceefa28bab42e0adcb2b6345c27a62 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4037cbe8f1de462ca703b3580eeb1af5, 60ceefa28bab42e0adcb2b6345c27a62; no voters:
I20260925 15:12:59.143776 14585 raft_consensus.cc:2781] T 57d7907070e14f2c9fb6528cb7355784 P 60ceefa28bab42e0adcb2b6345c27a62 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:12:59.143893 14585 raft_consensus.cc:482] T 57d7907070e14f2c9fb6528cb7355784 P 60ceefa28bab42e0adcb2b6345c27a62 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:12:59.143983 14585 raft_consensus.cc:3037] T 57d7907070e14f2c9fb6528cb7355784 P 60ceefa28bab42e0adcb2b6345c27a62 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:12:59.147207 14585 raft_consensus.cc:504] T 57d7907070e14f2c9fb6528cb7355784 P 60ceefa28bab42e0adcb2b6345c27a62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ceefa28bab42e0adcb2b6345c27a62" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 34225 } } peers { permanent_uuid: "4037cbe8f1de462ca703b3580eeb1af5" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 43891 } } peers { permanent_uuid: "3e607a2588f04ef7bd2c71fb7752782d" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43695 } }
I20260925 15:12:59.148274 14376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57d7907070e14f2c9fb6528cb7355784" candidate_uuid: "60ceefa28bab42e0adcb2b6345c27a62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4037cbe8f1de462ca703b3580eeb1af5"
I20260925 15:12:59.148344 14514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57d7907070e14f2c9fb6528cb7355784" candidate_uuid: "60ceefa28bab42e0adcb2b6345c27a62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e607a2588f04ef7bd2c71fb7752782d"
I20260925 15:12:59.148483 14376 raft_consensus.cc:3037] T 57d7907070e14f2c9fb6528cb7355784 P 4037cbe8f1de462ca703b3580eeb1af5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:12:59.148526 14514 raft_consensus.cc:3037] T 57d7907070e14f2c9fb6528cb7355784 P 3e607a2588f04ef7bd2c71fb7752782d [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:12:59.151434 14514 raft_consensus.cc:2454] T 57d7907070e14f2c9fb6528cb7355784 P 3e607a2588f04ef7bd2c71fb7752782d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60ceefa28bab42e0adcb2b6345c27a62 in term 1.
I20260925 15:12:59.151856 14163 leader_election.cc:302] T 57d7907070e14f2c9fb6528cb7355784 P 60ceefa28bab42e0adcb2b6345c27a62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e607a2588f04ef7bd2c71fb7752782d, 60ceefa28bab42e0adcb2b6345c27a62; no voters:
I20260925 15:12:59.152149 14613 raft_consensus.cc:2781] T 57d7907070e14f2c9fb6528cb7355784 P 60ceefa28bab42e0adcb2b6345c27a62 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:12:59.152287 14376 raft_consensus.cc:2454] T 57d7907070e14f2c9fb6528cb7355784 P 4037cbe8f1de462ca703b3580eeb1af5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60ceefa28bab42e0adcb2b6345c27a62 in term 1.
I20260925 15:12:59.152292 14613 raft_consensus.cc:686] T 57d7907070e14f2c9fb6528cb7355784 P 60ceefa28bab42e0adcb2b6345c27a62 [term 1 LEADER]: Becoming Leader. State: Replica: 60ceefa28bab42e0adcb2b6345c27a62, State: Running, Role: LEADER
I20260925 15:12:59.152649 14613 consensus_queue.cc:237] T 57d7907070e14f2c9fb6528cb7355784 P 60ceefa28bab42e0adcb2b6345c27a62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ceefa28bab42e0adcb2b6345c27a62" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 34225 } } peers { permanent_uuid: "4037cbe8f1de462ca703b3580eeb1af5" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 43891 } } peers { permanent_uuid: "3e607a2588f04ef7bd2c71fb7752782d" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43695 } }
I20260925 15:12:59.154145 14585 leader_election.cc:288] T 57d7907070e14f2c9fb6528cb7355784 P 60ceefa28bab42e0adcb2b6345c27a62 [CANDIDATE]: Term 1 election: Requested vote from peers 4037cbe8f1de462ca703b3580eeb1af5 (127.12.8.2:43891), 3e607a2588f04ef7bd2c71fb7752782d (127.12.8.3:43695)
I20260925 15:12:59.156126 14075 catalog_manager.cc:5495] T 57d7907070e14f2c9fb6528cb7355784 P 60ceefa28bab42e0adcb2b6345c27a62 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60ceefa28bab42e0adcb2b6345c27a62 (127.12.8.1). New cstate: current_term: 1 leader_uuid: "60ceefa28bab42e0adcb2b6345c27a62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ceefa28bab42e0adcb2b6345c27a62" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 34225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4037cbe8f1de462ca703b3580eeb1af5" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 43891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e607a2588f04ef7bd2c71fb7752782d" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43695 } health_report { overall_health: UNKNOWN } } }
I20260925 15:12:59.198693 14574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:59.192253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48982 (local address 127.12.8.1:34225)
0925 15:12:59.192459 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:59.192465 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:59.192484 (+ 19us) server_negotiation.cc:407] Connection header received
0925 15:12:59.192621 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:59.192626 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:59.192705 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:59.192800 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:12:59.193236 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:59.194421 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:59.196425 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:59.197154 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:59.197274 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:59.197836 (+ 562us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:12:59.197994 (+ 158us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:12:59.198098 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:59.198280 (+ 182us) server_negotiation.cc:299] Negotiation successful
0925 15:12:59.198363 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 15:12:59.214710 14574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:59.208027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.1:54833 (local address 127.12.8.1:34225)
0925 15:12:59.208599 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:59.208605 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:59.208624 (+ 19us) server_negotiation.cc:407] Connection header received
0925 15:12:59.208674 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:59.208679 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:59.208749 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:59.208837 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:12:59.209287 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:59.210500 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:59.212566 (+ 2066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:59.213281 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:59.213408 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:59.213875 (+ 467us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:12:59.214019 (+ 144us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:12:59.214112 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:59.214290 (+ 178us) server_negotiation.cc:299] Negotiation successful
0925 15:12:59.214394 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":462}
I20260925 15:12:59.214773 14625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:59.207966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.1:34225 (local address 127.12.8.1:54833)
0925 15:12:59.208225 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:59.208246 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 15:12:59.208378 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:12:59.208879 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:12:59.208885 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:12:59.208898 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 15:12:59.209138 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:59.209152 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:59.210663 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:59.210670 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:59.212385 (+ 1715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:59.212399 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:59.213434 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:59.213441 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:59.213648 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:59.213692 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 15:12:59.214133 (+ 441us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 15:12:59.214145 (+ 12us) client_negotiation.cc:769] Sending connection context
0925 15:12:59.214254 (+ 109us) client_negotiation.cc:240] Negotiation successful
0925 15:12:59.214424 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"mutex_wait_us":44}
I20260925 15:12:59.225545 14514 raft_consensus.cc:1261] T 57d7907070e14f2c9fb6528cb7355784 P 3e607a2588f04ef7bd2c71fb7752782d [term 1 FOLLOWER]: Refusing update from remote peer 60ceefa28bab42e0adcb2b6345c27a62: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:12:59.225555 14376 raft_consensus.cc:1261] T 57d7907070e14f2c9fb6528cb7355784 P 4037cbe8f1de462ca703b3580eeb1af5 [term 1 FOLLOWER]: Refusing update from remote peer 60ceefa28bab42e0adcb2b6345c27a62: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:12:59.226096 14613 consensus_queue.cc:1035] T 57d7907070e14f2c9fb6528cb7355784 P 60ceefa28bab42e0adcb2b6345c27a62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e607a2588f04ef7bd2c71fb7752782d" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:12:59.226354 14603 consensus_queue.cc:1035] T 57d7907070e14f2c9fb6528cb7355784 P 60ceefa28bab42e0adcb2b6345c27a62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4037cbe8f1de462ca703b3580eeb1af5" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 43891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:12:59.239440 14608 mvcc.cc:205] Tried to move back new op lower bound from 7333270238103031808 to 7333270237812039680. Current Snapshot: MvccSnapshot[applied={T|T < 7333270238103031808}]
I20260925 15:12:59.240474 14610 mvcc.cc:205] Tried to move back new op lower bound from 7333270238103031808 to 7333270237812039680. Current Snapshot: MvccSnapshot[applied={T|T < 7333270238103031808}]
W20260925 15:12:59.260609 14608 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 15:13:00.854765 14074 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42528:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:13:00.854990 14074 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42528:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:13:00.858906 14074 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8ee449b6924246da81d3c6ac299728f6: Sending DeleteTablet for 3 replicas of tablet 57d7907070e14f2c9fb6528cb7355784
I20260925 15:13:00.859639 14211 tablet_service.cc:1514] Processing DeleteTablet for tablet 57d7907070e14f2c9fb6528cb7355784 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:13:00 UTC) from {username='kudu', principal='kudu/127.12.8.62@KRBTEST.COM'} at 127.0.0.1:48958
I20260925 15:13:00.859825 14350 tablet_service.cc:1514] Processing DeleteTablet for tablet 57d7907070e14f2c9fb6528cb7355784 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:13:00 UTC) from {username='kudu', principal='kudu/127.12.8.62@KRBTEST.COM'} at 127.0.0.1:49170
I20260925 15:13:00.860003 14489 tablet_service.cc:1514] Processing DeleteTablet for tablet 57d7907070e14f2c9fb6528cb7355784 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:13:00 UTC) from {username='kudu', principal='kudu/127.12.8.62@KRBTEST.COM'} at 127.0.0.1:60456
I20260925 15:13:00.860823 14660 tablet_replica.cc:330] T 57d7907070e14f2c9fb6528cb7355784 P 3e607a2588f04ef7bd2c71fb7752782d: stopping tablet replica
I20260925 15:13:00.860978 14659 tablet_replica.cc:330] T 57d7907070e14f2c9fb6528cb7355784 P 4037cbe8f1de462ca703b3580eeb1af5: stopping tablet replica
I20260925 15:13:00.861176 14659 raft_consensus.cc:2229] T 57d7907070e14f2c9fb6528cb7355784 P 4037cbe8f1de462ca703b3580eeb1af5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:13:00.861164 14660 raft_consensus.cc:2229] T 57d7907070e14f2c9fb6528cb7355784 P 3e607a2588f04ef7bd2c71fb7752782d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:13:00.861433 14659 raft_consensus.cc:2258] T 57d7907070e14f2c9fb6528cb7355784 P 4037cbe8f1de462ca703b3580eeb1af5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:13:00.861483 14660 raft_consensus.cc:2258] T 57d7907070e14f2c9fb6528cb7355784 P 3e607a2588f04ef7bd2c71fb7752782d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:13:00.862449 14658 tablet_replica.cc:330] T 57d7907070e14f2c9fb6528cb7355784 P 60ceefa28bab42e0adcb2b6345c27a62: stopping tablet replica
I20260925 15:13:00.862643 14658 raft_consensus.cc:2229] T 57d7907070e14f2c9fb6528cb7355784 P 60ceefa28bab42e0adcb2b6345c27a62 [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:13:00.862784 14660 ts_tablet_manager.cc:1905] T 57d7907070e14f2c9fb6528cb7355784 P 3e607a2588f04ef7bd2c71fb7752782d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:13:00.863209 14658 raft_consensus.cc:2258] T 57d7907070e14f2c9fb6528cb7355784 P 60ceefa28bab42e0adcb2b6345c27a62 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:13:00.864085 14659 ts_tablet_manager.cc:1905] T 57d7907070e14f2c9fb6528cb7355784 P 4037cbe8f1de462ca703b3580eeb1af5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:13:00.864168 14658 ts_tablet_manager.cc:1905] T 57d7907070e14f2c9fb6528cb7355784 P 60ceefa28bab42e0adcb2b6345c27a62: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:13:00.870237 14660 ts_tablet_manager.cc:1918] T 57d7907070e14f2c9fb6528cb7355784 P 3e607a2588f04ef7bd2c71fb7752782d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 15:13:00.870440 14660 log.cc:1196] T 57d7907070e14f2c9fb6528cb7355784 P 3e607a2588f04ef7bd2c71fb7752782d: Deleting WAL directory at /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/minicluster-data/ts-2/wal/wals/57d7907070e14f2c9fb6528cb7355784
I20260925 15:13:00.870935 14660 ts_tablet_manager.cc:1939] T 57d7907070e14f2c9fb6528cb7355784 P 3e607a2588f04ef7bd2c71fb7752782d: Deleting consensus metadata
I20260925 15:13:00.871134 14659 ts_tablet_manager.cc:1918] T 57d7907070e14f2c9fb6528cb7355784 P 4037cbe8f1de462ca703b3580eeb1af5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 15:13:00.871269 14659 log.cc:1196] T 57d7907070e14f2c9fb6528cb7355784 P 4037cbe8f1de462ca703b3580eeb1af5: Deleting WAL directory at /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/minicluster-data/ts-1/wal/wals/57d7907070e14f2c9fb6528cb7355784
I20260925 15:13:00.871268 14658 ts_tablet_manager.cc:1918] T 57d7907070e14f2c9fb6528cb7355784 P 60ceefa28bab42e0adcb2b6345c27a62: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 15:13:00.871395 14658 log.cc:1196] T 57d7907070e14f2c9fb6528cb7355784 P 60ceefa28bab42e0adcb2b6345c27a62: Deleting WAL directory at /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/minicluster-data/ts-0/wal/wals/57d7907070e14f2c9fb6528cb7355784
I20260925 15:13:00.871759 14659 ts_tablet_manager.cc:1939] T 57d7907070e14f2c9fb6528cb7355784 P 4037cbe8f1de462ca703b3580eeb1af5: Deleting consensus metadata
I20260925 15:13:00.871899 14658 ts_tablet_manager.cc:1939] T 57d7907070e14f2c9fb6528cb7355784 P 60ceefa28bab42e0adcb2b6345c27a62: Deleting consensus metadata
I20260925 15:13:00.872589 14060 catalog_manager.cc:4841] TS 3e607a2588f04ef7bd2c71fb7752782d (127.12.8.3:43695): tablet 57d7907070e14f2c9fb6528cb7355784 (table test-table [id=83eca47c4bc44f6ebfede9ec9c3b0bc9]) successfully deleted
I20260925 15:13:00.872822 14059 catalog_manager.cc:4841] TS 4037cbe8f1de462ca703b3580eeb1af5 (127.12.8.2:43891): tablet 57d7907070e14f2c9fb6528cb7355784 (table test-table [id=83eca47c4bc44f6ebfede9ec9c3b0bc9]) successfully deleted
I20260925 15:13:00.873062 14060 catalog_manager.cc:4841] TS 60ceefa28bab42e0adcb2b6345c27a62 (127.12.8.1:34225): tablet 57d7907070e14f2c9fb6528cb7355784 (table test-table [id=83eca47c4bc44f6ebfede9ec9c3b0bc9]) successfully deleted
Sep 25 15:13:00 dist-test-slave-p468 krb5kdc[14030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349180, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 15:13:00 dist-test-slave-p468 krb5kdc[14030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349180, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
I20260925 15:13:00.911337 14668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:00.897675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42536 (local address 127.12.8.62:33113)
0925 15:13:00.898270 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:00.898274 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:00.898299 (+ 25us) server_negotiation.cc:407] Connection header received
0925 15:13:00.898467 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:00.898471 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:00.898531 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:00.898623 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:00.899883 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:00.900633 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:00.901928 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:00.902387 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:00.902489 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:00.906301 (+ 3812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:00.906306 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:00.906310 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:00.906341 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:00.908928 (+ 2587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:00.909682 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:00.909686 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:00.909688 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:00.909814 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:00.910102 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:00.910105 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:00.910107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:00.910441 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:00.910607 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:00.910935 (+ 328us) server_negotiation.cc:299] Negotiation successful
0925 15:13:00.911088 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":262,"threads_started":1}
I20260925 15:13:00.914923 14074 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42536:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:13:00.915493 14074 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:13:00.921736 14350 tablet_service.cc:1467] Processing CreateTablet for tablet 106c589ac7fe44c18ee541a876645163 (DEFAULT_TABLE table=test-table [id=491a9967507a4cdeb784d9131e0fea15]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:13:00.921739 14211 tablet_service.cc:1467] Processing CreateTablet for tablet 106c589ac7fe44c18ee541a876645163 (DEFAULT_TABLE table=test-table [id=491a9967507a4cdeb784d9131e0fea15]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:13:00.921897 14489 tablet_service.cc:1467] Processing CreateTablet for tablet 106c589ac7fe44c18ee541a876645163 (DEFAULT_TABLE table=test-table [id=491a9967507a4cdeb784d9131e0fea15]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:13:00.922235 14211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 106c589ac7fe44c18ee541a876645163. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:13:00.922241 14350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 106c589ac7fe44c18ee541a876645163. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:13:00.922302 14489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 106c589ac7fe44c18ee541a876645163. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:13:00.928992 14670 tablet_bootstrap.cc:492] T 106c589ac7fe44c18ee541a876645163 P 3e607a2588f04ef7bd2c71fb7752782d: Bootstrap starting.
I20260925 15:13:00.929774 14669 tablet_bootstrap.cc:492] T 106c589ac7fe44c18ee541a876645163 P 60ceefa28bab42e0adcb2b6345c27a62: Bootstrap starting.
I20260925 15:13:00.930994 14670 tablet_bootstrap.cc:654] T 106c589ac7fe44c18ee541a876645163 P 3e607a2588f04ef7bd2c71fb7752782d: Neither blocks nor log segments found. Creating new log.
I20260925 15:13:00.932005 14669 tablet_bootstrap.cc:654] T 106c589ac7fe44c18ee541a876645163 P 60ceefa28bab42e0adcb2b6345c27a62: Neither blocks nor log segments found. Creating new log.
I20260925 15:13:00.932305 14671 tablet_bootstrap.cc:492] T 106c589ac7fe44c18ee541a876645163 P 4037cbe8f1de462ca703b3580eeb1af5: Bootstrap starting.
I20260925 15:13:00.933753 14671 tablet_bootstrap.cc:654] T 106c589ac7fe44c18ee541a876645163 P 4037cbe8f1de462ca703b3580eeb1af5: Neither blocks nor log segments found. Creating new log.
I20260925 15:13:00.933954 14669 tablet_bootstrap.cc:492] T 106c589ac7fe44c18ee541a876645163 P 60ceefa28bab42e0adcb2b6345c27a62: No bootstrap required, opened a new log
I20260925 15:13:00.934100 14669 ts_tablet_manager.cc:1397] T 106c589ac7fe44c18ee541a876645163 P 60ceefa28bab42e0adcb2b6345c27a62: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260925 15:13:00.934969 14669 raft_consensus.cc:348] T 106c589ac7fe44c18ee541a876645163 P 60ceefa28bab42e0adcb2b6345c27a62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ceefa28bab42e0adcb2b6345c27a62" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 34225 } } peers { permanent_uuid: "4037cbe8f1de462ca703b3580eeb1af5" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 43891 } } peers { permanent_uuid: "3e607a2588f04ef7bd2c71fb7752782d" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43695 } }
I20260925 15:13:00.935225 14669 raft_consensus.cc:374] T 106c589ac7fe44c18ee541a876645163 P 60ceefa28bab42e0adcb2b6345c27a62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:13:00.935335 14669 raft_consensus.cc:729] T 106c589ac7fe44c18ee541a876645163 P 60ceefa28bab42e0adcb2b6345c27a62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ceefa28bab42e0adcb2b6345c27a62, State: Initialized, Role: FOLLOWER
I20260925 15:13:00.935706 14670 tablet_bootstrap.cc:492] T 106c589ac7fe44c18ee541a876645163 P 3e607a2588f04ef7bd2c71fb7752782d: No bootstrap required, opened a new log
I20260925 15:13:00.935599 14669 consensus_queue.cc:260] T 106c589ac7fe44c18ee541a876645163 P 60ceefa28bab42e0adcb2b6345c27a62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ceefa28bab42e0adcb2b6345c27a62" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 34225 } } peers { permanent_uuid: "4037cbe8f1de462ca703b3580eeb1af5" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 43891 } } peers { permanent_uuid: "3e607a2588f04ef7bd2c71fb7752782d" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43695 } }
I20260925 15:13:00.935853 14670 ts_tablet_manager.cc:1397] T 106c589ac7fe44c18ee541a876645163 P 3e607a2588f04ef7bd2c71fb7752782d: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260925 15:13:00.936280 14669 ts_tablet_manager.cc:1428] T 106c589ac7fe44c18ee541a876645163 P 60ceefa28bab42e0adcb2b6345c27a62: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 15:13:00.936501 14670 raft_consensus.cc:348] T 106c589ac7fe44c18ee541a876645163 P 3e607a2588f04ef7bd2c71fb7752782d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ceefa28bab42e0adcb2b6345c27a62" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 34225 } } peers { permanent_uuid: "4037cbe8f1de462ca703b3580eeb1af5" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 43891 } } peers { permanent_uuid: "3e607a2588f04ef7bd2c71fb7752782d" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43695 } }
I20260925 15:13:00.936722 14670 raft_consensus.cc:374] T 106c589ac7fe44c18ee541a876645163 P 3e607a2588f04ef7bd2c71fb7752782d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:13:00.936833 14670 raft_consensus.cc:729] T 106c589ac7fe44c18ee541a876645163 P 3e607a2588f04ef7bd2c71fb7752782d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e607a2588f04ef7bd2c71fb7752782d, State: Initialized, Role: FOLLOWER
I20260925 15:13:00.937074 14670 consensus_queue.cc:260] T 106c589ac7fe44c18ee541a876645163 P 3e607a2588f04ef7bd2c71fb7752782d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ceefa28bab42e0adcb2b6345c27a62" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 34225 } } peers { permanent_uuid: "4037cbe8f1de462ca703b3580eeb1af5" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 43891 } } peers { permanent_uuid: "3e607a2588f04ef7bd2c71fb7752782d" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43695 } }
I20260925 15:13:00.938076 14671 tablet_bootstrap.cc:492] T 106c589ac7fe44c18ee541a876645163 P 4037cbe8f1de462ca703b3580eeb1af5: No bootstrap required, opened a new log
I20260925 15:13:00.938264 14671 ts_tablet_manager.cc:1397] T 106c589ac7fe44c18ee541a876645163 P 4037cbe8f1de462ca703b3580eeb1af5: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260925 15:13:00.939039 14671 raft_consensus.cc:348] T 106c589ac7fe44c18ee541a876645163 P 4037cbe8f1de462ca703b3580eeb1af5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ceefa28bab42e0adcb2b6345c27a62" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 34225 } } peers { permanent_uuid: "4037cbe8f1de462ca703b3580eeb1af5" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 43891 } } peers { permanent_uuid: "3e607a2588f04ef7bd2c71fb7752782d" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43695 } }
I20260925 15:13:00.939296 14671 raft_consensus.cc:374] T 106c589ac7fe44c18ee541a876645163 P 4037cbe8f1de462ca703b3580eeb1af5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:13:00.939395 14671 raft_consensus.cc:729] T 106c589ac7fe44c18ee541a876645163 P 4037cbe8f1de462ca703b3580eeb1af5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4037cbe8f1de462ca703b3580eeb1af5, State: Initialized, Role: FOLLOWER
I20260925 15:13:00.939620 14671 consensus_queue.cc:260] T 106c589ac7fe44c18ee541a876645163 P 4037cbe8f1de462ca703b3580eeb1af5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ceefa28bab42e0adcb2b6345c27a62" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 34225 } } peers { permanent_uuid: "4037cbe8f1de462ca703b3580eeb1af5" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 43891 } } peers { permanent_uuid: "3e607a2588f04ef7bd2c71fb7752782d" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43695 } }
I20260925 15:13:00.940292 14670 ts_tablet_manager.cc:1428] T 106c589ac7fe44c18ee541a876645163 P 3e607a2588f04ef7bd2c71fb7752782d: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260925 15:13:00.940799 14671 ts_tablet_manager.cc:1428] T 106c589ac7fe44c18ee541a876645163 P 4037cbe8f1de462ca703b3580eeb1af5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 15:13:01.027273 14641 raft_consensus.cc:482] T 106c589ac7fe44c18ee541a876645163 P 60ceefa28bab42e0adcb2b6345c27a62 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:13:01.027519 14641 raft_consensus.cc:504] T 106c589ac7fe44c18ee541a876645163 P 60ceefa28bab42e0adcb2b6345c27a62 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ceefa28bab42e0adcb2b6345c27a62" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 34225 } } peers { permanent_uuid: "4037cbe8f1de462ca703b3580eeb1af5" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 43891 } } peers { permanent_uuid: "3e607a2588f04ef7bd2c71fb7752782d" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43695 } }
I20260925 15:13:01.028277 14641 leader_election.cc:288] T 106c589ac7fe44c18ee541a876645163 P 60ceefa28bab42e0adcb2b6345c27a62 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4037cbe8f1de462ca703b3580eeb1af5 (127.12.8.2:43891), 3e607a2588f04ef7bd2c71fb7752782d (127.12.8.3:43695)
I20260925 15:13:01.028784 14376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106c589ac7fe44c18ee541a876645163" candidate_uuid: "60ceefa28bab42e0adcb2b6345c27a62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4037cbe8f1de462ca703b3580eeb1af5" is_pre_election: true
I20260925 15:13:01.028821 14515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106c589ac7fe44c18ee541a876645163" candidate_uuid: "60ceefa28bab42e0adcb2b6345c27a62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e607a2588f04ef7bd2c71fb7752782d" is_pre_election: true
I20260925 15:13:01.029012 14376 raft_consensus.cc:2454] T 106c589ac7fe44c18ee541a876645163 P 4037cbe8f1de462ca703b3580eeb1af5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60ceefa28bab42e0adcb2b6345c27a62 in term 0.
I20260925 15:13:01.029066 14515 raft_consensus.cc:2454] T 106c589ac7fe44c18ee541a876645163 P 3e607a2588f04ef7bd2c71fb7752782d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60ceefa28bab42e0adcb2b6345c27a62 in term 0.
I20260925 15:13:01.029537 14162 leader_election.cc:302] T 106c589ac7fe44c18ee541a876645163 P 60ceefa28bab42e0adcb2b6345c27a62 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4037cbe8f1de462ca703b3580eeb1af5, 60ceefa28bab42e0adcb2b6345c27a62; no voters:
I20260925 15:13:01.029824 14641 raft_consensus.cc:2781] T 106c589ac7fe44c18ee541a876645163 P 60ceefa28bab42e0adcb2b6345c27a62 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:13:01.029948 14641 raft_consensus.cc:482] T 106c589ac7fe44c18ee541a876645163 P 60ceefa28bab42e0adcb2b6345c27a62 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:13:01.030063 14641 raft_consensus.cc:3037] T 106c589ac7fe44c18ee541a876645163 P 60ceefa28bab42e0adcb2b6345c27a62 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:13:01.032115 14641 raft_consensus.cc:504] T 106c589ac7fe44c18ee541a876645163 P 60ceefa28bab42e0adcb2b6345c27a62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ceefa28bab42e0adcb2b6345c27a62" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 34225 } } peers { permanent_uuid: "4037cbe8f1de462ca703b3580eeb1af5" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 43891 } } peers { permanent_uuid: "3e607a2588f04ef7bd2c71fb7752782d" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43695 } }
I20260925 15:13:01.032625 14641 leader_election.cc:288] T 106c589ac7fe44c18ee541a876645163 P 60ceefa28bab42e0adcb2b6345c27a62 [CANDIDATE]: Term 1 election: Requested vote from peers 4037cbe8f1de462ca703b3580eeb1af5 (127.12.8.2:43891), 3e607a2588f04ef7bd2c71fb7752782d (127.12.8.3:43695)
I20260925 15:13:01.033188 14515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106c589ac7fe44c18ee541a876645163" candidate_uuid: "60ceefa28bab42e0adcb2b6345c27a62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e607a2588f04ef7bd2c71fb7752782d"
I20260925 15:13:01.033195 14376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106c589ac7fe44c18ee541a876645163" candidate_uuid: "60ceefa28bab42e0adcb2b6345c27a62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4037cbe8f1de462ca703b3580eeb1af5"
I20260925 15:13:01.033398 14515 raft_consensus.cc:3037] T 106c589ac7fe44c18ee541a876645163 P 3e607a2588f04ef7bd2c71fb7752782d [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:13:01.033393 14376 raft_consensus.cc:3037] T 106c589ac7fe44c18ee541a876645163 P 4037cbe8f1de462ca703b3580eeb1af5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:13:01.036383 14376 raft_consensus.cc:2454] T 106c589ac7fe44c18ee541a876645163 P 4037cbe8f1de462ca703b3580eeb1af5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60ceefa28bab42e0adcb2b6345c27a62 in term 1.
I20260925 15:13:01.036383 14515 raft_consensus.cc:2454] T 106c589ac7fe44c18ee541a876645163 P 3e607a2588f04ef7bd2c71fb7752782d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60ceefa28bab42e0adcb2b6345c27a62 in term 1.
I20260925 15:13:01.036784 14162 leader_election.cc:302] T 106c589ac7fe44c18ee541a876645163 P 60ceefa28bab42e0adcb2b6345c27a62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4037cbe8f1de462ca703b3580eeb1af5, 60ceefa28bab42e0adcb2b6345c27a62; no voters:
I20260925 15:13:01.037096 14641 raft_consensus.cc:2781] T 106c589ac7fe44c18ee541a876645163 P 60ceefa28bab42e0adcb2b6345c27a62 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:13:01.037210 14641 raft_consensus.cc:686] T 106c589ac7fe44c18ee541a876645163 P 60ceefa28bab42e0adcb2b6345c27a62 [term 1 LEADER]: Becoming Leader. State: Replica: 60ceefa28bab42e0adcb2b6345c27a62, State: Running, Role: LEADER
I20260925 15:13:01.037504 14641 consensus_queue.cc:237] T 106c589ac7fe44c18ee541a876645163 P 60ceefa28bab42e0adcb2b6345c27a62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ceefa28bab42e0adcb2b6345c27a62" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 34225 } } peers { permanent_uuid: "4037cbe8f1de462ca703b3580eeb1af5" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 43891 } } peers { permanent_uuid: "3e607a2588f04ef7bd2c71fb7752782d" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43695 } }
I20260925 15:13:01.040159 14074 catalog_manager.cc:5495] T 106c589ac7fe44c18ee541a876645163 P 60ceefa28bab42e0adcb2b6345c27a62 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60ceefa28bab42e0adcb2b6345c27a62 (127.12.8.1). New cstate: current_term: 1 leader_uuid: "60ceefa28bab42e0adcb2b6345c27a62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ceefa28bab42e0adcb2b6345c27a62" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 34225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4037cbe8f1de462ca703b3580eeb1af5" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 43891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e607a2588f04ef7bd2c71fb7752782d" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43695 } health_report { overall_health: UNKNOWN } } }
I20260925 15:13:01.070703 14679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:01.063939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48990 (local address 127.12.8.1:34225)
0925 15:13:01.064482 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:01.064487 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:01.064505 (+ 18us) server_negotiation.cc:407] Connection header received
0925 15:13:01.064552 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:01.064556 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:01.064616 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:01.064694 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:13:01.065209 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:01.066069 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:01.068671 (+ 2602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:01.069165 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:01.069315 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:01.069923 (+ 608us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:13:01.070038 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:13:01.070140 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:01.070319 (+ 179us) server_negotiation.cc:299] Negotiation successful
0925 15:13:01.070380 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":345,"threads_started":1}
I20260925 15:13:01.079905 14515 raft_consensus.cc:1261] T 106c589ac7fe44c18ee541a876645163 P 3e607a2588f04ef7bd2c71fb7752782d [term 1 FOLLOWER]: Refusing update from remote peer 60ceefa28bab42e0adcb2b6345c27a62: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:13:01.079926 14376 raft_consensus.cc:1261] T 106c589ac7fe44c18ee541a876645163 P 4037cbe8f1de462ca703b3580eeb1af5 [term 1 FOLLOWER]: Refusing update from remote peer 60ceefa28bab42e0adcb2b6345c27a62: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:13:01.080462 14646 consensus_queue.cc:1035] T 106c589ac7fe44c18ee541a876645163 P 60ceefa28bab42e0adcb2b6345c27a62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e607a2588f04ef7bd2c71fb7752782d" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:13:01.080919 14646 consensus_queue.cc:1035] T 106c589ac7fe44c18ee541a876645163 P 60ceefa28bab42e0adcb2b6345c27a62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4037cbe8f1de462ca703b3580eeb1af5" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 43891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:13:01.131549 14074 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42536:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:13:01.131855 14074 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42536:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:13:01.135264 14074 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8ee449b6924246da81d3c6ac299728f6: Sending DeleteTablet for 3 replicas of tablet 106c589ac7fe44c18ee541a876645163
I20260925 15:13:01.136228 14489 tablet_service.cc:1514] Processing DeleteTablet for tablet 106c589ac7fe44c18ee541a876645163 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:13:01 UTC) from {username='kudu', principal='kudu/127.12.8.62@KRBTEST.COM'} at 127.0.0.1:60456
I20260925 15:13:01.136246 14350 tablet_service.cc:1514] Processing DeleteTablet for tablet 106c589ac7fe44c18ee541a876645163 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:13:01 UTC) from {username='kudu', principal='kudu/127.12.8.62@KRBTEST.COM'} at 127.0.0.1:49170
I20260925 15:13:01.136507 14660 tablet_replica.cc:330] T 106c589ac7fe44c18ee541a876645163 P 3e607a2588f04ef7bd2c71fb7752782d: stopping tablet replica
I20260925 15:13:01.136451 14211 tablet_service.cc:1514] Processing DeleteTablet for tablet 106c589ac7fe44c18ee541a876645163 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:13:01 UTC) from {username='kudu', principal='kudu/127.12.8.62@KRBTEST.COM'} at 127.0.0.1:48958
I20260925 15:13:01.136529 14659 tablet_replica.cc:330] T 106c589ac7fe44c18ee541a876645163 P 4037cbe8f1de462ca703b3580eeb1af5: stopping tablet replica
I20260925 15:13:01.136713 14660 raft_consensus.cc:2229] T 106c589ac7fe44c18ee541a876645163 P 3e607a2588f04ef7bd2c71fb7752782d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:13:01.136724 14659 raft_consensus.cc:2229] T 106c589ac7fe44c18ee541a876645163 P 4037cbe8f1de462ca703b3580eeb1af5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:13:01.136855 14660 raft_consensus.cc:2258] T 106c589ac7fe44c18ee541a876645163 P 3e607a2588f04ef7bd2c71fb7752782d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:13:01.136904 14659 raft_consensus.cc:2258] T 106c589ac7fe44c18ee541a876645163 P 4037cbe8f1de462ca703b3580eeb1af5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:13:01.137125 14658 tablet_replica.cc:330] T 106c589ac7fe44c18ee541a876645163 P 60ceefa28bab42e0adcb2b6345c27a62: stopping tablet replica
I20260925 15:13:01.137336 14658 raft_consensus.cc:2229] T 106c589ac7fe44c18ee541a876645163 P 60ceefa28bab42e0adcb2b6345c27a62 [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:13:01.137657 14658 raft_consensus.cc:2258] T 106c589ac7fe44c18ee541a876645163 P 60ceefa28bab42e0adcb2b6345c27a62 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:13:01.138180 14659 ts_tablet_manager.cc:1905] T 106c589ac7fe44c18ee541a876645163 P 4037cbe8f1de462ca703b3580eeb1af5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:13:01.138484 14074 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42536:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 15:13:01.138938 14658 ts_tablet_manager.cc:1905] T 106c589ac7fe44c18ee541a876645163 P 60ceefa28bab42e0adcb2b6345c27a62: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 15:13:01.139084 14074 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:13:01.140890 14660 ts_tablet_manager.cc:1905] T 106c589ac7fe44c18ee541a876645163 P 3e607a2588f04ef7bd2c71fb7752782d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:13:01.145126 14659 ts_tablet_manager.cc:1918] T 106c589ac7fe44c18ee541a876645163 P 4037cbe8f1de462ca703b3580eeb1af5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 15:13:01.145272 14659 log.cc:1196] T 106c589ac7fe44c18ee541a876645163 P 4037cbe8f1de462ca703b3580eeb1af5: Deleting WAL directory at /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/minicluster-data/ts-1/wal/wals/106c589ac7fe44c18ee541a876645163
I20260925 15:13:01.145744 14659 ts_tablet_manager.cc:1939] T 106c589ac7fe44c18ee541a876645163 P 4037cbe8f1de462ca703b3580eeb1af5: Deleting consensus metadata
I20260925 15:13:01.146824 14059 catalog_manager.cc:4841] TS 4037cbe8f1de462ca703b3580eeb1af5 (127.12.8.2:43891): tablet 106c589ac7fe44c18ee541a876645163 (table test-table [id=491a9967507a4cdeb784d9131e0fea15]) successfully deleted
I20260925 15:13:01.148130 14660 ts_tablet_manager.cc:1918] T 106c589ac7fe44c18ee541a876645163 P 3e607a2588f04ef7bd2c71fb7752782d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 15:13:01.148269 14660 log.cc:1196] T 106c589ac7fe44c18ee541a876645163 P 3e607a2588f04ef7bd2c71fb7752782d: Deleting WAL directory at /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/minicluster-data/ts-2/wal/wals/106c589ac7fe44c18ee541a876645163
I20260925 15:13:01.148583 14660 ts_tablet_manager.cc:1939] T 106c589ac7fe44c18ee541a876645163 P 3e607a2588f04ef7bd2c71fb7752782d: Deleting consensus metadata
I20260925 15:13:01.149504 14060 catalog_manager.cc:4841] TS 3e607a2588f04ef7bd2c71fb7752782d (127.12.8.3:43695): tablet 106c589ac7fe44c18ee541a876645163 (table test-table [id=491a9967507a4cdeb784d9131e0fea15]) successfully deleted
I20260925 15:13:01.153303 14658 ts_tablet_manager.cc:1918] T 106c589ac7fe44c18ee541a876645163 P 60ceefa28bab42e0adcb2b6345c27a62: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 15:13:01.153481 14658 log.cc:1196] T 106c589ac7fe44c18ee541a876645163 P 60ceefa28bab42e0adcb2b6345c27a62: Deleting WAL directory at /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/minicluster-data/ts-0/wal/wals/106c589ac7fe44c18ee541a876645163
I20260925 15:13:01.153990 14658 ts_tablet_manager.cc:1939] T 106c589ac7fe44c18ee541a876645163 P 60ceefa28bab42e0adcb2b6345c27a62: Deleting consensus metadata
I20260925 15:13:01.154827 14211 tablet_service.cc:1467] Processing CreateTablet for tablet 6018efe4a8f64899b23cf20e2cb7ae48 (DEFAULT_TABLE table=test-table [id=aea7e66f095e4affb50f91f5f4f96e5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:13:01.155280 14211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6018efe4a8f64899b23cf20e2cb7ae48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:13:01.155233 14350 tablet_service.cc:1467] Processing CreateTablet for tablet 6018efe4a8f64899b23cf20e2cb7ae48 (DEFAULT_TABLE table=test-table [id=aea7e66f095e4affb50f91f5f4f96e5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:13:01.155651 14350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6018efe4a8f64899b23cf20e2cb7ae48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:13:01.158608 14060 catalog_manager.cc:4841] TS 60ceefa28bab42e0adcb2b6345c27a62 (127.12.8.1:34225): tablet 106c589ac7fe44c18ee541a876645163 (table test-table [id=491a9967507a4cdeb784d9131e0fea15]) successfully deleted
I20260925 15:13:01.159168 14489 tablet_service.cc:1467] Processing CreateTablet for tablet 6018efe4a8f64899b23cf20e2cb7ae48 (DEFAULT_TABLE table=test-table [id=aea7e66f095e4affb50f91f5f4f96e5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:13:01.159601 14489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6018efe4a8f64899b23cf20e2cb7ae48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:13:01.162680 14669 tablet_bootstrap.cc:492] T 6018efe4a8f64899b23cf20e2cb7ae48 P 60ceefa28bab42e0adcb2b6345c27a62: Bootstrap starting.
I20260925 15:13:01.164021 14669 tablet_bootstrap.cc:654] T 6018efe4a8f64899b23cf20e2cb7ae48 P 60ceefa28bab42e0adcb2b6345c27a62: Neither blocks nor log segments found. Creating new log.
I20260925 15:13:01.165095 14671 tablet_bootstrap.cc:492] T 6018efe4a8f64899b23cf20e2cb7ae48 P 4037cbe8f1de462ca703b3580eeb1af5: Bootstrap starting.
I20260925 15:13:01.166939 14671 tablet_bootstrap.cc:654] T 6018efe4a8f64899b23cf20e2cb7ae48 P 4037cbe8f1de462ca703b3580eeb1af5: Neither blocks nor log segments found. Creating new log.
I20260925 15:13:01.169770 14670 tablet_bootstrap.cc:492] T 6018efe4a8f64899b23cf20e2cb7ae48 P 3e607a2588f04ef7bd2c71fb7752782d: Bootstrap starting.
I20260925 15:13:01.172200 14670 tablet_bootstrap.cc:654] T 6018efe4a8f64899b23cf20e2cb7ae48 P 3e607a2588f04ef7bd2c71fb7752782d: Neither blocks nor log segments found. Creating new log.
I20260925 15:13:01.178928 14669 tablet_bootstrap.cc:492] T 6018efe4a8f64899b23cf20e2cb7ae48 P 60ceefa28bab42e0adcb2b6345c27a62: No bootstrap required, opened a new log
I20260925 15:13:01.179186 14669 ts_tablet_manager.cc:1397] T 6018efe4a8f64899b23cf20e2cb7ae48 P 60ceefa28bab42e0adcb2b6345c27a62: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.009s
I20260925 15:13:01.180215 14669 raft_consensus.cc:348] T 6018efe4a8f64899b23cf20e2cb7ae48 P 60ceefa28bab42e0adcb2b6345c27a62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ceefa28bab42e0adcb2b6345c27a62" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 34225 } } peers { permanent_uuid: "4037cbe8f1de462ca703b3580eeb1af5" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 43891 } } peers { permanent_uuid: "3e607a2588f04ef7bd2c71fb7752782d" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43695 } }
I20260925 15:13:01.180620 14669 raft_consensus.cc:374] T 6018efe4a8f64899b23cf20e2cb7ae48 P 60ceefa28bab42e0adcb2b6345c27a62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:13:01.180778 14669 raft_consensus.cc:729] T 6018efe4a8f64899b23cf20e2cb7ae48 P 60ceefa28bab42e0adcb2b6345c27a62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ceefa28bab42e0adcb2b6345c27a62, State: Initialized, Role: FOLLOWER
I20260925 15:13:01.181156 14669 consensus_queue.cc:260] T 6018efe4a8f64899b23cf20e2cb7ae48 P 60ceefa28bab42e0adcb2b6345c27a62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ceefa28bab42e0adcb2b6345c27a62" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 34225 } } peers { permanent_uuid: "4037cbe8f1de462ca703b3580eeb1af5" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 43891 } } peers { permanent_uuid: "3e607a2588f04ef7bd2c71fb7752782d" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43695 } }
I20260925 15:13:01.181980 14669 ts_tablet_manager.cc:1428] T 6018efe4a8f64899b23cf20e2cb7ae48 P 60ceefa28bab42e0adcb2b6345c27a62: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260925 15:13:01.182770 14671 tablet_bootstrap.cc:492] T 6018efe4a8f64899b23cf20e2cb7ae48 P 4037cbe8f1de462ca703b3580eeb1af5: No bootstrap required, opened a new log
I20260925 15:13:01.182977 14671 ts_tablet_manager.cc:1397] T 6018efe4a8f64899b23cf20e2cb7ae48 P 4037cbe8f1de462ca703b3580eeb1af5: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.007s
I20260925 15:13:01.183909 14670 tablet_bootstrap.cc:492] T 6018efe4a8f64899b23cf20e2cb7ae48 P 3e607a2588f04ef7bd2c71fb7752782d: No bootstrap required, opened a new log
I20260925 15:13:01.183982 14671 raft_consensus.cc:348] T 6018efe4a8f64899b23cf20e2cb7ae48 P 4037cbe8f1de462ca703b3580eeb1af5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ceefa28bab42e0adcb2b6345c27a62" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 34225 } } peers { permanent_uuid: "4037cbe8f1de462ca703b3580eeb1af5" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 43891 } } peers { permanent_uuid: "3e607a2588f04ef7bd2c71fb7752782d" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43695 } }
I20260925 15:13:01.184160 14670 ts_tablet_manager.cc:1397] T 6018efe4a8f64899b23cf20e2cb7ae48 P 3e607a2588f04ef7bd2c71fb7752782d: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.004s
I20260925 15:13:01.184273 14671 raft_consensus.cc:374] T 6018efe4a8f64899b23cf20e2cb7ae48 P 4037cbe8f1de462ca703b3580eeb1af5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:13:01.184371 14671 raft_consensus.cc:729] T 6018efe4a8f64899b23cf20e2cb7ae48 P 4037cbe8f1de462ca703b3580eeb1af5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4037cbe8f1de462ca703b3580eeb1af5, State: Initialized, Role: FOLLOWER
I20260925 15:13:01.184672 14671 consensus_queue.cc:260] T 6018efe4a8f64899b23cf20e2cb7ae48 P 4037cbe8f1de462ca703b3580eeb1af5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ceefa28bab42e0adcb2b6345c27a62" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 34225 } } peers { permanent_uuid: "4037cbe8f1de462ca703b3580eeb1af5" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 43891 } } peers { permanent_uuid: "3e607a2588f04ef7bd2c71fb7752782d" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43695 } }
I20260925 15:13:01.185371 14670 raft_consensus.cc:348] T 6018efe4a8f64899b23cf20e2cb7ae48 P 3e607a2588f04ef7bd2c71fb7752782d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ceefa28bab42e0adcb2b6345c27a62" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 34225 } } peers { permanent_uuid: "4037cbe8f1de462ca703b3580eeb1af5" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 43891 } } peers { permanent_uuid: "3e607a2588f04ef7bd2c71fb7752782d" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43695 } }
I20260925 15:13:01.185758 14670 raft_consensus.cc:374] T 6018efe4a8f64899b23cf20e2cb7ae48 P 3e607a2588f04ef7bd2c71fb7752782d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:13:01.185868 14670 raft_consensus.cc:729] T 6018efe4a8f64899b23cf20e2cb7ae48 P 3e607a2588f04ef7bd2c71fb7752782d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e607a2588f04ef7bd2c71fb7752782d, State: Initialized, Role: FOLLOWER
I20260925 15:13:01.186153 14670 consensus_queue.cc:260] T 6018efe4a8f64899b23cf20e2cb7ae48 P 3e607a2588f04ef7bd2c71fb7752782d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ceefa28bab42e0adcb2b6345c27a62" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 34225 } } peers { permanent_uuid: "4037cbe8f1de462ca703b3580eeb1af5" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 43891 } } peers { permanent_uuid: "3e607a2588f04ef7bd2c71fb7752782d" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43695 } }
I20260925 15:13:01.186910 14670 ts_tablet_manager.cc:1428] T 6018efe4a8f64899b23cf20e2cb7ae48 P 3e607a2588f04ef7bd2c71fb7752782d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260925 15:13:01.186472 14671 ts_tablet_manager.cc:1428] T 6018efe4a8f64899b23cf20e2cb7ae48 P 4037cbe8f1de462ca703b3580eeb1af5: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260925 15:13:01.307902 14638 raft_consensus.cc:482] T 6018efe4a8f64899b23cf20e2cb7ae48 P 60ceefa28bab42e0adcb2b6345c27a62 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:13:01.308234 14638 raft_consensus.cc:504] T 6018efe4a8f64899b23cf20e2cb7ae48 P 60ceefa28bab42e0adcb2b6345c27a62 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ceefa28bab42e0adcb2b6345c27a62" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 34225 } } peers { permanent_uuid: "4037cbe8f1de462ca703b3580eeb1af5" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 43891 } } peers { permanent_uuid: "3e607a2588f04ef7bd2c71fb7752782d" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43695 } }
I20260925 15:13:01.308898 14638 leader_election.cc:288] T 6018efe4a8f64899b23cf20e2cb7ae48 P 60ceefa28bab42e0adcb2b6345c27a62 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4037cbe8f1de462ca703b3580eeb1af5 (127.12.8.2:43891), 3e607a2588f04ef7bd2c71fb7752782d (127.12.8.3:43695)
I20260925 15:13:01.309538 14515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6018efe4a8f64899b23cf20e2cb7ae48" candidate_uuid: "60ceefa28bab42e0adcb2b6345c27a62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e607a2588f04ef7bd2c71fb7752782d" is_pre_election: true
I20260925 15:13:01.309849 14515 raft_consensus.cc:2454] T 6018efe4a8f64899b23cf20e2cb7ae48 P 3e607a2588f04ef7bd2c71fb7752782d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60ceefa28bab42e0adcb2b6345c27a62 in term 0.
I20260925 15:13:01.309794 14375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6018efe4a8f64899b23cf20e2cb7ae48" candidate_uuid: "60ceefa28bab42e0adcb2b6345c27a62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4037cbe8f1de462ca703b3580eeb1af5" is_pre_election: true
I20260925 15:13:01.310124 14375 raft_consensus.cc:2454] T 6018efe4a8f64899b23cf20e2cb7ae48 P 4037cbe8f1de462ca703b3580eeb1af5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60ceefa28bab42e0adcb2b6345c27a62 in term 0.
I20260925 15:13:01.310361 14163 leader_election.cc:302] T 6018efe4a8f64899b23cf20e2cb7ae48 P 60ceefa28bab42e0adcb2b6345c27a62 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e607a2588f04ef7bd2c71fb7752782d, 60ceefa28bab42e0adcb2b6345c27a62; no voters:
I20260925 15:13:01.310618 14638 raft_consensus.cc:2781] T 6018efe4a8f64899b23cf20e2cb7ae48 P 60ceefa28bab42e0adcb2b6345c27a62 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:13:01.310781 14638 raft_consensus.cc:482] T 6018efe4a8f64899b23cf20e2cb7ae48 P 60ceefa28bab42e0adcb2b6345c27a62 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:13:01.310885 14638 raft_consensus.cc:3037] T 6018efe4a8f64899b23cf20e2cb7ae48 P 60ceefa28bab42e0adcb2b6345c27a62 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:13:01.313051 14638 raft_consensus.cc:504] T 6018efe4a8f64899b23cf20e2cb7ae48 P 60ceefa28bab42e0adcb2b6345c27a62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ceefa28bab42e0adcb2b6345c27a62" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 34225 } } peers { permanent_uuid: "4037cbe8f1de462ca703b3580eeb1af5" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 43891 } } peers { permanent_uuid: "3e607a2588f04ef7bd2c71fb7752782d" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43695 } }
I20260925 15:13:01.313570 14638 leader_election.cc:288] T 6018efe4a8f64899b23cf20e2cb7ae48 P 60ceefa28bab42e0adcb2b6345c27a62 [CANDIDATE]: Term 1 election: Requested vote from peers 4037cbe8f1de462ca703b3580eeb1af5 (127.12.8.2:43891), 3e607a2588f04ef7bd2c71fb7752782d (127.12.8.3:43695)
I20260925 15:13:01.314047 14515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6018efe4a8f64899b23cf20e2cb7ae48" candidate_uuid: "60ceefa28bab42e0adcb2b6345c27a62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e607a2588f04ef7bd2c71fb7752782d"
I20260925 15:13:01.314080 14375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6018efe4a8f64899b23cf20e2cb7ae48" candidate_uuid: "60ceefa28bab42e0adcb2b6345c27a62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4037cbe8f1de462ca703b3580eeb1af5"
I20260925 15:13:01.314272 14515 raft_consensus.cc:3037] T 6018efe4a8f64899b23cf20e2cb7ae48 P 3e607a2588f04ef7bd2c71fb7752782d [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:13:01.314369 14375 raft_consensus.cc:3037] T 6018efe4a8f64899b23cf20e2cb7ae48 P 4037cbe8f1de462ca703b3580eeb1af5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:13:01.316409 14515 raft_consensus.cc:2454] T 6018efe4a8f64899b23cf20e2cb7ae48 P 3e607a2588f04ef7bd2c71fb7752782d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60ceefa28bab42e0adcb2b6345c27a62 in term 1.
I20260925 15:13:01.316818 14163 leader_election.cc:302] T 6018efe4a8f64899b23cf20e2cb7ae48 P 60ceefa28bab42e0adcb2b6345c27a62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e607a2588f04ef7bd2c71fb7752782d, 60ceefa28bab42e0adcb2b6345c27a62; no voters:
I20260925 15:13:01.317066 14638 raft_consensus.cc:2781] T 6018efe4a8f64899b23cf20e2cb7ae48 P 60ceefa28bab42e0adcb2b6345c27a62 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:13:01.317232 14638 raft_consensus.cc:686] T 6018efe4a8f64899b23cf20e2cb7ae48 P 60ceefa28bab42e0adcb2b6345c27a62 [term 1 LEADER]: Becoming Leader. State: Replica: 60ceefa28bab42e0adcb2b6345c27a62, State: Running, Role: LEADER
I20260925 15:13:01.317500 14638 consensus_queue.cc:237] T 6018efe4a8f64899b23cf20e2cb7ae48 P 60ceefa28bab42e0adcb2b6345c27a62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ceefa28bab42e0adcb2b6345c27a62" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 34225 } } peers { permanent_uuid: "4037cbe8f1de462ca703b3580eeb1af5" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 43891 } } peers { permanent_uuid: "3e607a2588f04ef7bd2c71fb7752782d" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43695 } }
I20260925 15:13:01.318171 14375 raft_consensus.cc:2454] T 6018efe4a8f64899b23cf20e2cb7ae48 P 4037cbe8f1de462ca703b3580eeb1af5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60ceefa28bab42e0adcb2b6345c27a62 in term 1.
I20260925 15:13:01.321971 14074 catalog_manager.cc:5495] T 6018efe4a8f64899b23cf20e2cb7ae48 P 60ceefa28bab42e0adcb2b6345c27a62 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60ceefa28bab42e0adcb2b6345c27a62 (127.12.8.1). New cstate: current_term: 1 leader_uuid: "60ceefa28bab42e0adcb2b6345c27a62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ceefa28bab42e0adcb2b6345c27a62" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 34225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4037cbe8f1de462ca703b3580eeb1af5" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 43891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e607a2588f04ef7bd2c71fb7752782d" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43695 } health_report { overall_health: UNKNOWN } } }
I20260925 15:13:01.377862 14375 raft_consensus.cc:1261] T 6018efe4a8f64899b23cf20e2cb7ae48 P 4037cbe8f1de462ca703b3580eeb1af5 [term 1 FOLLOWER]: Refusing update from remote peer 60ceefa28bab42e0adcb2b6345c27a62: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:13:01.378021 14515 raft_consensus.cc:1261] T 6018efe4a8f64899b23cf20e2cb7ae48 P 3e607a2588f04ef7bd2c71fb7752782d [term 1 FOLLOWER]: Refusing update from remote peer 60ceefa28bab42e0adcb2b6345c27a62: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:13:01.378438 14687 consensus_queue.cc:1035] T 6018efe4a8f64899b23cf20e2cb7ae48 P 60ceefa28bab42e0adcb2b6345c27a62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4037cbe8f1de462ca703b3580eeb1af5" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 43891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:13:01.378789 14641 consensus_queue.cc:1035] T 6018efe4a8f64899b23cf20e2cb7ae48 P 60ceefa28bab42e0adcb2b6345c27a62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e607a2588f04ef7bd2c71fb7752782d" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 15:13:01.399793 14639 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 15:13:02.983264 14075 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42536:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:13:02.983508 14075 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42536:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:13:02.987298 14075 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8ee449b6924246da81d3c6ac299728f6: Sending DeleteTablet for 3 replicas of tablet 6018efe4a8f64899b23cf20e2cb7ae48
I20260925 15:13:02.988215 14212 tablet_service.cc:1514] Processing DeleteTablet for tablet 6018efe4a8f64899b23cf20e2cb7ae48 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:13:02 UTC) from {username='kudu', principal='kudu/127.12.8.62@KRBTEST.COM'} at 127.0.0.1:48958
I20260925 15:13:02.988271 14350 tablet_service.cc:1514] Processing DeleteTablet for tablet 6018efe4a8f64899b23cf20e2cb7ae48 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:13:02 UTC) from {username='kudu', principal='kudu/127.12.8.62@KRBTEST.COM'} at 127.0.0.1:49170
I20260925 15:13:02.988621 14489 tablet_service.cc:1514] Processing DeleteTablet for tablet 6018efe4a8f64899b23cf20e2cb7ae48 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:13:02 UTC) from {username='kudu', principal='kudu/127.12.8.62@KRBTEST.COM'} at 127.0.0.1:60456
I20260925 15:13:02.988770 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 14146
I20260925 15:13:02.998605 14711 tablet_replica.cc:330] T 6018efe4a8f64899b23cf20e2cb7ae48 P 3e607a2588f04ef7bd2c71fb7752782d: stopping tablet replica
I20260925 15:13:02.998894 14711 raft_consensus.cc:2229] T 6018efe4a8f64899b23cf20e2cb7ae48 P 3e607a2588f04ef7bd2c71fb7752782d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:13:02.999280 14711 raft_consensus.cc:2258] T 6018efe4a8f64899b23cf20e2cb7ae48 P 3e607a2588f04ef7bd2c71fb7752782d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:13:03.001953 14711 ts_tablet_manager.cc:1905] T 6018efe4a8f64899b23cf20e2cb7ae48 P 3e607a2588f04ef7bd2c71fb7752782d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:13:03.002496 14709 tablet_replica.cc:330] T 6018efe4a8f64899b23cf20e2cb7ae48 P 60ceefa28bab42e0adcb2b6345c27a62: stopping tablet replica
I20260925 15:13:03.002696 14709 raft_consensus.cc:2229] T 6018efe4a8f64899b23cf20e2cb7ae48 P 60ceefa28bab42e0adcb2b6345c27a62 [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:13:03.002756 14710 tablet_replica.cc:330] T 6018efe4a8f64899b23cf20e2cb7ae48 P 4037cbe8f1de462ca703b3580eeb1af5: stopping tablet replica
I20260925 15:13:03.003042 14710 raft_consensus.cc:2229] T 6018efe4a8f64899b23cf20e2cb7ae48 P 4037cbe8f1de462ca703b3580eeb1af5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:13:03.003069 14709 raft_consensus.cc:2258] T 6018efe4a8f64899b23cf20e2cb7ae48 P 60ceefa28bab42e0adcb2b6345c27a62 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:13:03.003691 14710 raft_consensus.cc:2258] T 6018efe4a8f64899b23cf20e2cb7ae48 P 4037cbe8f1de462ca703b3580eeb1af5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:13:03.003942 14709 ts_tablet_manager.cc:1905] T 6018efe4a8f64899b23cf20e2cb7ae48 P 60ceefa28bab42e0adcb2b6345c27a62: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:13:03.004786 14710 ts_tablet_manager.cc:1905] T 6018efe4a8f64899b23cf20e2cb7ae48 P 4037cbe8f1de462ca703b3580eeb1af5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:13:03.009586 14711 ts_tablet_manager.cc:1918] T 6018efe4a8f64899b23cf20e2cb7ae48 P 3e607a2588f04ef7bd2c71fb7752782d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
Sep 25 15:13:03 dist-test-slave-p468 krb5kdc[14030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349180, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.8.1@KRBTEST.COM
I20260925 15:13:03.009742 14711 log.cc:1196] T 6018efe4a8f64899b23cf20e2cb7ae48 P 3e607a2588f04ef7bd2c71fb7752782d: Deleting WAL directory at /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/minicluster-data/ts-2/wal/wals/6018efe4a8f64899b23cf20e2cb7ae48
I20260925 15:13:03.010164 14711 ts_tablet_manager.cc:1939] T 6018efe4a8f64899b23cf20e2cb7ae48 P 3e607a2588f04ef7bd2c71fb7752782d: Deleting consensus metadata
I20260925 15:13:03.010914 14709 ts_tablet_manager.cc:1918] T 6018efe4a8f64899b23cf20e2cb7ae48 P 60ceefa28bab42e0adcb2b6345c27a62: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 15:13:03.011073 14709 log.cc:1196] T 6018efe4a8f64899b23cf20e2cb7ae48 P 60ceefa28bab42e0adcb2b6345c27a62: Deleting WAL directory at /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/minicluster-data/ts-0/wal/wals/6018efe4a8f64899b23cf20e2cb7ae48
I20260925 15:13:03.011350 14060 catalog_manager.cc:4841] TS 3e607a2588f04ef7bd2c71fb7752782d (127.12.8.3:43695): tablet 6018efe4a8f64899b23cf20e2cb7ae48 (table test-table [id=aea7e66f095e4affb50f91f5f4f96e5c]) successfully deleted
I20260925 15:13:03.011585 14709 ts_tablet_manager.cc:1939] T 6018efe4a8f64899b23cf20e2cb7ae48 P 60ceefa28bab42e0adcb2b6345c27a62: Deleting consensus metadata
I20260925 15:13:03.012281 14710 ts_tablet_manager.cc:1918] T 6018efe4a8f64899b23cf20e2cb7ae48 P 4037cbe8f1de462ca703b3580eeb1af5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 15:13:03.012435 14710 log.cc:1196] T 6018efe4a8f64899b23cf20e2cb7ae48 P 4037cbe8f1de462ca703b3580eeb1af5: Deleting WAL directory at /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349162370577-12320-0/minicluster-data/ts-1/wal/wals/6018efe4a8f64899b23cf20e2cb7ae48
I20260925 15:13:03.012704 14060 catalog_manager.cc:4841] TS 60ceefa28bab42e0adcb2b6345c27a62 (127.12.8.1:34225): tablet 6018efe4a8f64899b23cf20e2cb7ae48 (table test-table [id=aea7e66f095e4affb50f91f5f4f96e5c]) successfully deleted
I20260925 15:13:03.012887 14710 ts_tablet_manager.cc:1939] T 6018efe4a8f64899b23cf20e2cb7ae48 P 4037cbe8f1de462ca703b3580eeb1af5: Deleting consensus metadata
I20260925 15:13:03.013975 14059 catalog_manager.cc:4841] TS 4037cbe8f1de462ca703b3580eeb1af5 (127.12.8.2:43891): tablet 6018efe4a8f64899b23cf20e2cb7ae48 (table test-table [id=aea7e66f095e4affb50f91f5f4f96e5c]) successfully deleted
I20260925 15:13:03.017701 14713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:02.992030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49006 (local address 127.12.8.1:34225)
0925 15:13:02.998362 (+ 6332us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:02.998372 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:02.998401 (+ 29us) server_negotiation.cc:407] Connection header received
0925 15:13:02.998478 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:02.998486 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:02.998601 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:02.998750 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:03.001102 (+ 2352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:03.002761 (+ 1659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:03.004692 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:03.005367 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:03.005488 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:03.011148 (+ 5660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:03.011177 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:03.011199 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:03.011263 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:03.015062 (+ 3799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:03.015730 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:03.015752 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:03.015755 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:03.015895 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:03.016253 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:03.016258 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:03.016260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:03.016645 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:03.016885 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:03.017150 (+ 265us) server_negotiation.cc:299] Negotiation successful
0925 15:13:03.017322 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6137,"thread_start_us":5983,"threads_started":1}
W20260925 15:13:03.018384 14277 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49006
W20260925 15:13:03.019052 12320 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 14146: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 15:13:03.019268 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 14146
I20260925 15:13:03.044227 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 14286
Sep 25 15:13:03 dist-test-slave-p468 krb5kdc[14030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349180, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.8.2@KRBTEST.COM
I20260925 15:13:03.059356 14714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:03.044948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49184 (local address 127.12.8.2:43891)
0925 15:13:03.045446 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:03.045450 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:03.045463 (+ 13us) server_negotiation.cc:407] Connection header received
0925 15:13:03.045513 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:03.045517 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:03.045571 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:03.045685 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:03.047191 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:03.048431 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:03.049807 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:03.050359 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:03.050464 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:03.054333 (+ 3869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:03.054346 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:03.054361 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:03.054392 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:03.056930 (+ 2538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:03.057494 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:03.057499 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:03.057502 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:03.057611 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:03.057909 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:03.057912 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:03.057914 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:03.058418 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:03.058662 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:03.058937 (+ 275us) server_negotiation.cc:299] Negotiation successful
0925 15:13:03.059086 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":234,"threads_started":1}
W20260925 15:13:03.059980 14416 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49184
W20260925 15:13:03.060554 12320 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 14286: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 15:13:03.060700 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 14286
I20260925 15:13:03.081765 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 14425
Sep 25 15:13:03 dist-test-slave-p468 krb5kdc[14030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349180, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.8.3@KRBTEST.COM
I20260925 15:13:03.095837 14715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:03.082466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60470 (local address 127.12.8.3:43695)
0925 15:13:03.082964 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:03.082968 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:03.082983 (+ 15us) server_negotiation.cc:407] Connection header received
0925 15:13:03.083034 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:03.083038 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:03.083094 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:03.083181 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:03.084350 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:03.085075 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:03.086305 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:03.086714 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:03.086810 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:03.090861 (+ 4051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:03.090873 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:03.090888 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:03.090928 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:03.093497 (+ 2569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:03.094040 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:03.094046 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:03.094048 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:03.094169 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:03.094557 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:03.094560 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:03.094562 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:03.094867 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:03.095071 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:03.095304 (+ 233us) server_negotiation.cc:299] Negotiation successful
0925 15:13:03.095453 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":282,"threads_started":1}
W20260925 15:13:03.096463 14555 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60470
W20260925 15:13:03.097070 12320 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 14425: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 15:13:03.097231 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 14425
I20260925 15:13:03.114542 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 14043
I20260925 15:13:03.115641 14115 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:13:03.231737 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 14043
2026-09-25T15:13:03Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8140 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349162370577-12320-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:13:03 dist-test-slave-p468 krb5kdc[14719](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:13:03 dist-test-slave-p468 krb5kdc[14719](info): set up 2 sockets
Sep 25 15:13:03 dist-test-slave-p468 krb5kdc[14719](info): commencing operation
krb5kdc: starting...
W20260925 15:13:04.326821 12320 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.004s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349162370577-12320-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349162370577-12320-0/krb5kdc/test-user.keytab.
Sep 25 15:13:04 dist-test-slave-p468 krb5kdc[14719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349184, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:13:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:13:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.8.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab.
Entry for principal kudu/127.12.8.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab.
I20260925 15:13:04.509720 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349162370577-12320-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349162370577-12320-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349162370577-12320-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.62:35663
--webserver_interface=127.12.8.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab
--principal=kudu/127.12.8.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:46305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.62:35663
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:13:04.820571 14732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:13:04.820922 14732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:13:04.821094 14732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:13:04.828248 14732 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:13:04.828384 14732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:13:04.828482 14732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:13:04.828588 14732 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:13:04.828662 14732 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:13:04.838151 14732 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:46305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349162370577-12320-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349162370577-12320-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.8.62:35663
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab
--principal=kudu/127.12.8.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.62:35663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.8.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349162370577-12320-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:13:04.839429 14732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:13:04.840732 14732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:13:04.853631 14740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:04.853699 14737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:04.853746 14738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:13:04.859031 14732 server_base.cc:1034] running on GCE node
I20260925 15:13:04.859795 14732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:13:04.861132 14732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:13:04.862351 14732 hybrid_clock.cc:648] HybridClock initialized: now 1790349184862310 us; error 58 us; skew 500 ppm
I20260925 15:13:04.862676 14732 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:13:04 dist-test-slave-p468 krb5kdc[14719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349184, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:13:04.865614 14732 init.cc:377] Logged in from keytab as kudu/127.12.8.62@KRBTEST.COM (short username kudu)
I20260925 15:13:04.867085 14732 webserver.cc:460] Webserver started at http://127.12.8.62:36537/ using document root <none> and password file <none>
I20260925 15:13:04.867765 14732 fs_manager.cc:359] Metadata directory not provided
I20260925 15:13:04.867904 14732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:13:04.868194 14732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:13:04.871002 14732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349162370577-12320-0/minicluster-data/master-0/data/instance:
uuid: "f42dab5666954142b9c43b385e7eeded"
format_stamp: "Formatted at 2026-09-25 15:13:04 on dist-test-slave-p468"
server_key: "794076f2bd8d8d3cfe93361ef84bae6c"
server_key_iv: "6d1e01ae1499635fede8704ab014bddc"
server_key_version: "encryptionkey@0"
I20260925 15:13:04.871727 14732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349162370577-12320-0/minicluster-data/master-0/wal/instance:
uuid: "f42dab5666954142b9c43b385e7eeded"
format_stamp: "Formatted at 2026-09-25 15:13:04 on dist-test-slave-p468"
server_key: "794076f2bd8d8d3cfe93361ef84bae6c"
server_key_iv: "6d1e01ae1499635fede8704ab014bddc"
server_key_version: "encryptionkey@0"
I20260925 15:13:04.876611 14732 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260925 15:13:04.880064 14747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:04.881366 14732 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260925 15:13:04.881542 14732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349162370577-12320-0/minicluster-data/master-0/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349162370577-12320-0/minicluster-data/master-0/wal
uuid: "f42dab5666954142b9c43b385e7eeded"
format_stamp: "Formatted at 2026-09-25 15:13:04 on dist-test-slave-p468"
server_key: "794076f2bd8d8d3cfe93361ef84bae6c"
server_key_iv: "6d1e01ae1499635fede8704ab014bddc"
server_key_version: "encryptionkey@0"
I20260925 15:13:04.881733 14732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349162370577-12320-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349162370577-12320-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349162370577-12320-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:13:04.899957 14732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:13:04.904207 14732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:13:04.904484 14732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:13:04.922698 14732 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.62:35663
I20260925 15:13:04.922778 14798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.62:35663 every 8 connection(s)
I20260925 15:13:04.924050 14732 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
I20260925 15:13:04.925069 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 14732
I20260925 15:13:04.925292 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349162370577-12320-0/minicluster-data/master-0/wal/instance
I20260925 15:13:04.925840 12320 external_mini_cluster.cc:1428] Setting key 536a5cd897a7a716d4b91c34d2618446
I20260925 15:13:04.928369 14799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 25 15:13:04 dist-test-slave-p468 krb5kdc[14719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349184, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
I20260925 15:13:04.940884 14799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f42dab5666954142b9c43b385e7eeded: Bootstrap starting.
I20260925 15:13:04.944079 14799 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f42dab5666954142b9c43b385e7eeded: Neither blocks nor log segments found. Creating new log.
I20260925 15:13:04.945230 14799 log.cc:824] T 00000000000000000000000000000000 P f42dab5666954142b9c43b385e7eeded: Log is configured to *not* fsync() on all Append() calls
I20260925 15:13:04.948052 14799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f42dab5666954142b9c43b385e7eeded: No bootstrap required, opened a new log
I20260925 15:13:04.948732 14802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:04.927738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33734 (local address 127.12.8.62:35663)
0925 15:13:04.928503 (+ 765us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:04.928519 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:04.928567 (+ 48us) server_negotiation.cc:407] Connection header received
0925 15:13:04.930911 (+ 2344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:04.930936 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:04.931440 (+ 504us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:04.931847 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:04.933104 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:04.935197 (+ 2093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:04.936404 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:04.937400 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:04.937596 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:04.941619 (+ 4023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:04.941627 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:04.941640 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:04.941676 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:04.944696 (+ 3020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:04.945501 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:04.945507 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:04.945513 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:04.945663 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:04.946039 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:04.946045 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:04.946048 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:04.946598 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:04.946864 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:04.947293 (+ 429us) server_negotiation.cc:299] Negotiation successful
0925 15:13:04.947695 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":298,"threads_started":1}
I20260925 15:13:04.955814 14799 raft_consensus.cc:348] T 00000000000000000000000000000000 P f42dab5666954142b9c43b385e7eeded [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f42dab5666954142b9c43b385e7eeded" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 35663 } }
I20260925 15:13:04.956156 14799 raft_consensus.cc:374] T 00000000000000000000000000000000 P f42dab5666954142b9c43b385e7eeded [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:13:04.956264 14799 raft_consensus.cc:729] T 00000000000000000000000000000000 P f42dab5666954142b9c43b385e7eeded [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f42dab5666954142b9c43b385e7eeded, State: Initialized, Role: FOLLOWER
I20260925 15:13:04.956753 14799 consensus_queue.cc:260] T 00000000000000000000000000000000 P f42dab5666954142b9c43b385e7eeded [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f42dab5666954142b9c43b385e7eeded" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 35663 } }
I20260925 15:13:04.956979 14799 raft_consensus.cc:388] T 00000000000000000000000000000000 P f42dab5666954142b9c43b385e7eeded [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:13:04.957114 14799 raft_consensus.cc:482] T 00000000000000000000000000000000 P f42dab5666954142b9c43b385e7eeded [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:13:04.957288 14799 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f42dab5666954142b9c43b385e7eeded [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:13:04.959357 14799 raft_consensus.cc:504] T 00000000000000000000000000000000 P f42dab5666954142b9c43b385e7eeded [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f42dab5666954142b9c43b385e7eeded" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 35663 } }
I20260925 15:13:04.959863 14799 leader_election.cc:302] T 00000000000000000000000000000000 P f42dab5666954142b9c43b385e7eeded [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f42dab5666954142b9c43b385e7eeded; no voters:
I20260925 15:13:04.960376 14799 leader_election.cc:288] T 00000000000000000000000000000000 P f42dab5666954142b9c43b385e7eeded [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:13:04.960500 14804 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f42dab5666954142b9c43b385e7eeded [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:13:04.961748 14804 raft_consensus.cc:686] T 00000000000000000000000000000000 P f42dab5666954142b9c43b385e7eeded [term 1 LEADER]: Becoming Leader. State: Replica: f42dab5666954142b9c43b385e7eeded, State: Running, Role: LEADER
I20260925 15:13:04.962334 14804 consensus_queue.cc:237] T 00000000000000000000000000000000 P f42dab5666954142b9c43b385e7eeded [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f42dab5666954142b9c43b385e7eeded" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 35663 } }
I20260925 15:13:04.962527 14799 sys_catalog.cc:564] T 00000000000000000000000000000000 P f42dab5666954142b9c43b385e7eeded [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:13:04.965464 14805 sys_catalog.cc:455] T 00000000000000000000000000000000 P f42dab5666954142b9c43b385e7eeded [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f42dab5666954142b9c43b385e7eeded" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f42dab5666954142b9c43b385e7eeded" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 35663 } } }
I20260925 15:13:04.965869 14805 sys_catalog.cc:458] T 00000000000000000000000000000000 P f42dab5666954142b9c43b385e7eeded [sys.catalog]: This master's current role is: LEADER
I20260925 15:13:04.966310 14806 sys_catalog.cc:455] T 00000000000000000000000000000000 P f42dab5666954142b9c43b385e7eeded [sys.catalog]: SysCatalogTable state changed. Reason: New leader f42dab5666954142b9c43b385e7eeded. Latest consensus state: current_term: 1 leader_uuid: "f42dab5666954142b9c43b385e7eeded" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f42dab5666954142b9c43b385e7eeded" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 35663 } } }
I20260925 15:13:04.966578 14806 sys_catalog.cc:458] T 00000000000000000000000000000000 P f42dab5666954142b9c43b385e7eeded [sys.catalog]: This master's current role is: LEADER
I20260925 15:13:04.970888 14813 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:13:04.975225 14813 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:13:04.983822 14813 catalog_manager.cc:1347] Generated new cluster ID: 0d987707c7464e87b547c0e110a7d9b3
I20260925 15:13:04.983942 14813 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:13:05.023993 14813 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:13:05.025269 14813 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:13:05.034648 14813 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f42dab5666954142b9c43b385e7eeded: Generated new TSK 0
I20260925 15:13:05.035449 14813 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.8.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.1.keytab.
Entry for principal kudu/127.12.8.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.1.keytab.
I20260925 15:13:05.082006 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349162370577-12320-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349162370577-12320-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349162370577-12320-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349162370577-12320-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.1:0
--local_ip_for_outbound_sockets=127.12.8.1
--webserver_interface=127.12.8.1
--webserver_port=0
--tserver_master_addrs=127.12.8.62:35663
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.1.keytab
--principal=kudu/127.12.8.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:46305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:13:05.393102 14825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:13:05.393371 14825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:13:05.393461 14825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:13:05.400451 14825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:13:05.400545 14825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:13:05.400658 14825 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.1
I20260925 15:13:05.409781 14825 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:46305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349162370577-12320-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349162370577-12320-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.1.keytab
--principal=kudu/127.12.8.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349162370577-12320-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.8.1
--webserver_port=0
--tserver_master_addrs=127.12.8.62:35663
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.1
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349162370577-12320-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:13:05.410984 14825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:13:05.412207 14825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:13:05.424887 14830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:05.424928 14833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:05.425050 14831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:13:05.425591 14825 server_base.cc:1034] running on GCE node
I20260925 15:13:05.426128 14825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:13:05.427012 14825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:13:05.428239 14825 hybrid_clock.cc:648] HybridClock initialized: now 1790349185428184 us; error 47 us; skew 500 ppm
I20260925 15:13:05.428526 14825 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:13:05 dist-test-slave-p468 krb5kdc[14719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349185, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:13:05.431591 14825 init.cc:377] Logged in from keytab as kudu/127.12.8.1@KRBTEST.COM (short username kudu)
I20260925 15:13:05.432947 14825 webserver.cc:460] Webserver started at http://127.12.8.1:37233/ using document root <none> and password file <none>
I20260925 15:13:05.433575 14825 fs_manager.cc:359] Metadata directory not provided
I20260925 15:13:05.433717 14825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:13:05.434005 14825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:13:05.436779 14825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349162370577-12320-0/minicluster-data/ts-0/data/instance:
uuid: "c02994e40b61476a8442baa808e0144b"
format_stamp: "Formatted at 2026-09-25 15:13:05 on dist-test-slave-p468"
server_key: "ed7c3c8798516c79ba64bc3081fbe632"
server_key_iv: "c09eea953ab78b54370d2d094c6ab4c2"
server_key_version: "encryptionkey@0"
I20260925 15:13:05.437546 14825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349162370577-12320-0/minicluster-data/ts-0/wal/instance:
uuid: "c02994e40b61476a8442baa808e0144b"
format_stamp: "Formatted at 2026-09-25 15:13:05 on dist-test-slave-p468"
server_key: "ed7c3c8798516c79ba64bc3081fbe632"
server_key_iv: "c09eea953ab78b54370d2d094c6ab4c2"
server_key_version: "encryptionkey@0"
I20260925 15:13:05.442126 14825 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 15:13:05.445222 14840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:05.446409 14825 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 15:13:05.446570 14825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349162370577-12320-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349162370577-12320-0/minicluster-data/ts-0/wal
uuid: "c02994e40b61476a8442baa808e0144b"
format_stamp: "Formatted at 2026-09-25 15:13:05 on dist-test-slave-p468"
server_key: "ed7c3c8798516c79ba64bc3081fbe632"
server_key_iv: "c09eea953ab78b54370d2d094c6ab4c2"
server_key_version: "encryptionkey@0"
I20260925 15:13:05.446766 14825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349162370577-12320-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349162370577-12320-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349162370577-12320-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:13:05.460659 14825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:13:05.464789 14825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:13:05.465108 14825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:13:05.466042 14825 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:13:05.467595 14825 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:13:05.467669 14825 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:05.467785 14825 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:13:05.467847 14825 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:05.500131 14825 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.1:33193
I20260925 15:13:05.500177 14952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.1:33193 every 8 connection(s)
I20260925 15:13:05.501377 14825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349162370577-12320-0/minicluster-data/ts-0/data/info.pb
I20260925 15:13:05.508548 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 14825
I20260925 15:13:05.508780 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349162370577-12320-0/minicluster-data/ts-0/wal/instance
I20260925 15:13:05.509531 12320 external_mini_cluster.cc:1428] Setting key c75616adb27b4653904e961aabd1cc18
Sep 25 15:13:05 dist-test-slave-p468 krb5kdc[14719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349185, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.1@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
I20260925 15:13:05.520592 14955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:05.503893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.1:58641 (local address 127.12.8.62:35663)
0925 15:13:05.504312 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:05.504316 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:05.505286 (+ 970us) server_negotiation.cc:407] Connection header received
0925 15:13:05.506973 (+ 1687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:05.506977 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:05.507039 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:05.507132 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:05.508523 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:05.509714 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:05.511202 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:05.511595 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:05.511748 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:05.515241 (+ 3493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:05.515246 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:05.515249 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:05.515278 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:05.517711 (+ 2433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:05.518475 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:05.518480 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:05.518482 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:05.518591 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:05.518999 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:05.519004 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:05.519007 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:05.519260 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:05.519410 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:05.520034 (+ 624us) server_negotiation.cc:299] Negotiation successful
0925 15:13:05.520237 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":211,"threads_started":1}
I20260925 15:13:05.521488 14956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:05.504454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:35663 (local address 127.12.8.1:58641)
0925 15:13:05.505106 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:05.505144 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:05.506736 (+ 1592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:05.507331 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:05.507338 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:05.507818 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:13:05.508356 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:05.508371 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:05.509848 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:05.509852 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:05.511051 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:05.511062 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:05.511826 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:05.511836 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:05.512112 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:05.512783 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:13:05.512804 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:13:05.515037 (+ 2233us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:13:05.517852 (+ 2815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:13:05.517860 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:13:05.517874 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:13:05.518350 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:13:05.518697 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:13:05.518704 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:13:05.518709 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:13:05.518886 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:13:05.519383 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:13:05.519390 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:13:05.519706 (+ 316us) client_negotiation.cc:769] Sending connection context
0925 15:13:05.519948 (+ 242us) client_negotiation.cc:240] Negotiation successful
0925 15:13:05.520432 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":265,"threads_started":1}
I20260925 15:13:05.523025 14953 heartbeater.cc:347] Connected to a master server at 127.12.8.62:35663
I20260925 15:13:05.523324 14953 heartbeater.cc:464] Registering TS with master...
I20260925 15:13:05.523983 14953 heartbeater.cc:511] Master 127.12.8.62:35663 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.12.8.2@KRBTEST.COM; defaulting to no policy
I20260925 15:13:05.526216 14764 ts_manager.cc:194] Registered new tserver with Master: c02994e40b61476a8442baa808e0144b (127.12.8.1:33193)
I20260925 15:13:05.527709 14764 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.1@KRBTEST.COM'} at 127.12.8.1:58641
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.2.keytab.
Entry for principal kudu/127.12.8.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.2.keytab.
I20260925 15:13:05.549805 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349162370577-12320-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349162370577-12320-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349162370577-12320-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349162370577-12320-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.2:0
--local_ip_for_outbound_sockets=127.12.8.2
--webserver_interface=127.12.8.2
--webserver_port=0
--tserver_master_addrs=127.12.8.62:35663
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.2.keytab
--principal=kudu/127.12.8.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:46305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:13:05.865605 14959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:13:05.865940 14959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:13:05.866050 14959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:13:05.873242 14959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:13:05.873337 14959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:13:05.873443 14959 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.2
I20260925 15:13:05.882977 14959 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:46305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349162370577-12320-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349162370577-12320-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.2.keytab
--principal=kudu/127.12.8.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349162370577-12320-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.8.2
--webserver_port=0
--tserver_master_addrs=127.12.8.62:35663
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.2
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349162370577-12320-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:13:05.884184 14959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:13:05.885442 14959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:13:05.898362 14965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:05.898386 14964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:05.898319 14967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:13:05.899734 14959 server_base.cc:1034] running on GCE node
I20260925 15:13:05.900313 14959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:13:05.901263 14959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:13:05.902437 14959 hybrid_clock.cc:648] HybridClock initialized: now 1790349185902375 us; error 52 us; skew 500 ppm
I20260925 15:13:05.902729 14959 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:13:05 dist-test-slave-p468 krb5kdc[14719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349185, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:13:05.905802 14959 init.cc:377] Logged in from keytab as kudu/127.12.8.2@KRBTEST.COM (short username kudu)
I20260925 15:13:05.907210 14959 webserver.cc:460] Webserver started at http://127.12.8.2:37063/ using document root <none> and password file <none>
I20260925 15:13:05.907899 14959 fs_manager.cc:359] Metadata directory not provided
I20260925 15:13:05.908035 14959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:13:05.908368 14959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:13:05.911423 14959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349162370577-12320-0/minicluster-data/ts-1/data/instance:
uuid: "4ecb753f6a18403f8e6caebdc3c38990"
format_stamp: "Formatted at 2026-09-25 15:13:05 on dist-test-slave-p468"
server_key: "0b779e675a8d989d90cc51b5259bcbc9"
server_key_iv: "ece45dd26d2a9e8fefff03244a560629"
server_key_version: "encryptionkey@0"
I20260925 15:13:05.912247 14959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349162370577-12320-0/minicluster-data/ts-1/wal/instance:
uuid: "4ecb753f6a18403f8e6caebdc3c38990"
format_stamp: "Formatted at 2026-09-25 15:13:05 on dist-test-slave-p468"
server_key: "0b779e675a8d989d90cc51b5259bcbc9"
server_key_iv: "ece45dd26d2a9e8fefff03244a560629"
server_key_version: "encryptionkey@0"
I20260925 15:13:05.917434 14959 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260925 15:13:05.920910 14974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:05.922317 14959 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:13:05.922463 14959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349162370577-12320-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349162370577-12320-0/minicluster-data/ts-1/wal
uuid: "4ecb753f6a18403f8e6caebdc3c38990"
format_stamp: "Formatted at 2026-09-25 15:13:05 on dist-test-slave-p468"
server_key: "0b779e675a8d989d90cc51b5259bcbc9"
server_key_iv: "ece45dd26d2a9e8fefff03244a560629"
server_key_version: "encryptionkey@0"
I20260925 15:13:05.922676 14959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349162370577-12320-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349162370577-12320-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349162370577-12320-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:13:05.943725 14959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:13:05.948169 14959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:13:05.948455 14959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:13:05.949333 14959 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:13:05.950666 14959 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:13:05.950779 14959 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:05.950929 14959 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:13:05.951001 14959 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:05.984323 14959 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.2:36617
I20260925 15:13:05.984372 15086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.2:36617 every 8 connection(s)
I20260925 15:13:05.985725 14959 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349162370577-12320-0/minicluster-data/ts-1/data/info.pb
I20260925 15:13:05.995971 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 14959
I20260925 15:13:05.996191 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349162370577-12320-0/minicluster-data/ts-1/wal/instance
I20260925 15:13:05.996624 12320 external_mini_cluster.cc:1428] Setting key 215db44d70a7b2b7bae67b9f0fb1e1e3
Sep 25 15:13:05 dist-test-slave-p468 krb5kdc[14719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349185, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.2@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
I20260925 15:13:06.006724 14955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:05.988617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.2:40255 (local address 127.12.8.62:35663)
0925 15:13:05.988808 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:05.988813 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:05.990085 (+ 1272us) server_negotiation.cc:407] Connection header received
0925 15:13:05.991945 (+ 1860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:05.991951 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:05.992039 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:05.992150 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:05.993645 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:05.994430 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:05.996000 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:05.996501 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:05.996628 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:06.000246 (+ 3618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:06.000256 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:06.000263 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:06.000308 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:06.003813 (+ 3505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:06.004495 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:06.004502 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:06.004510 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:06.004644 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:06.005070 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:06.005081 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:06.005084 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:06.005330 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:06.005483 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:06.006140 (+ 657us) server_negotiation.cc:299] Negotiation successful
0925 15:13:06.006374 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 15:13:06.007687 15089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:05.989158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:35663 (local address 127.12.8.2:40255)
0925 15:13:05.989901 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:05.989941 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:05.991655 (+ 1714us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:05.992392 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:05.992401 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:05.992930 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:13:05.993480 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:05.993496 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:05.994585 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:05.994589 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:05.995859 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:05.995871 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:05.996738 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:05.996745 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:05.996975 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:05.997630 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:13:05.997664 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:13:06.000035 (+ 2371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:13:06.003941 (+ 3906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:13:06.003947 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:13:06.003960 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:13:06.004370 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:13:06.004761 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:13:06.004767 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:13:06.004770 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:13:06.004951 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:13:06.005454 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:13:06.005462 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:13:06.005808 (+ 346us) client_negotiation.cc:769] Sending connection context
0925 15:13:06.006078 (+ 270us) client_negotiation.cc:240] Negotiation successful
0925 15:13:06.006617 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":287,"threads_started":1}
I20260925 15:13:06.009076 15087 heartbeater.cc:347] Connected to a master server at 127.12.8.62:35663
I20260925 15:13:06.009410 15087 heartbeater.cc:464] Registering TS with master...
I20260925 15:13:06.010332 15087 heartbeater.cc:511] Master 127.12.8.62:35663 requested a full tablet report, sending...
I20260925 15:13:06.012169 14764 ts_manager.cc:194] Registered new tserver with Master: 4ecb753f6a18403f8e6caebdc3c38990 (127.12.8.2:36617)
I20260925 15:13:06.013489 14764 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.2@KRBTEST.COM'} at 127.12.8.2:40255
WARNING: no policy specified for kudu/127.12.8.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.3.keytab.
Entry for principal kudu/127.12.8.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.3.keytab.
I20260925 15:13:06.041720 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349162370577-12320-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349162370577-12320-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349162370577-12320-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349162370577-12320-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.3:0
--local_ip_for_outbound_sockets=127.12.8.3
--webserver_interface=127.12.8.3
--webserver_port=0
--tserver_master_addrs=127.12.8.62:35663
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.3.keytab
--principal=kudu/127.12.8.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:46305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:13:06.351908 15092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:13:06.352183 15092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:13:06.352275 15092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:13:06.359251 15092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:13:06.359351 15092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:13:06.359467 15092 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.3
I20260925 15:13:06.369145 15092 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:46305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349162370577-12320-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349162370577-12320-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.3.keytab
--principal=kudu/127.12.8.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349162370577-12320-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.8.3
--webserver_port=0
--tserver_master_addrs=127.12.8.62:35663
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.3
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349162370577-12320-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:13:06.370344 15092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:13:06.371556 15092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:13:06.384884 15100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:06.384837 15097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:06.384927 15098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:13:06.386260 15092 server_base.cc:1034] running on GCE node
I20260925 15:13:06.386922 15092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:13:06.387809 15092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:13:06.389029 15092 hybrid_clock.cc:648] HybridClock initialized: now 1790349186388992 us; error 50 us; skew 500 ppm
I20260925 15:13:06.389326 15092 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:13:06 dist-test-slave-p468 krb5kdc[14719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349186, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:13:06.392390 15092 init.cc:377] Logged in from keytab as kudu/127.12.8.3@KRBTEST.COM (short username kudu)
I20260925 15:13:06.393703 15092 webserver.cc:460] Webserver started at http://127.12.8.3:40775/ using document root <none> and password file <none>
I20260925 15:13:06.394408 15092 fs_manager.cc:359] Metadata directory not provided
I20260925 15:13:06.394546 15092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:13:06.394846 15092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:13:06.397609 15092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349162370577-12320-0/minicluster-data/ts-2/data/instance:
uuid: "5cfd3cb798ef4c2d8d872b4ff8f465ab"
format_stamp: "Formatted at 2026-09-25 15:13:06 on dist-test-slave-p468"
server_key: "719ef68ec7ec87e202daa054a27158de"
server_key_iv: "9f3d55b8e8c38f64f4e1b27a64f0ca1d"
server_key_version: "encryptionkey@0"
I20260925 15:13:06.398429 15092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349162370577-12320-0/minicluster-data/ts-2/wal/instance:
uuid: "5cfd3cb798ef4c2d8d872b4ff8f465ab"
format_stamp: "Formatted at 2026-09-25 15:13:06 on dist-test-slave-p468"
server_key: "719ef68ec7ec87e202daa054a27158de"
server_key_iv: "9f3d55b8e8c38f64f4e1b27a64f0ca1d"
server_key_version: "encryptionkey@0"
I20260925 15:13:06.403143 15092 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 15:13:06.406267 15107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:06.407467 15092 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 15:13:06.407634 15092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349162370577-12320-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349162370577-12320-0/minicluster-data/ts-2/wal
uuid: "5cfd3cb798ef4c2d8d872b4ff8f465ab"
format_stamp: "Formatted at 2026-09-25 15:13:06 on dist-test-slave-p468"
server_key: "719ef68ec7ec87e202daa054a27158de"
server_key_iv: "9f3d55b8e8c38f64f4e1b27a64f0ca1d"
server_key_version: "encryptionkey@0"
I20260925 15:13:06.407841 15092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349162370577-12320-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349162370577-12320-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349162370577-12320-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:13:06.418778 15092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:13:06.422794 15092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:13:06.423061 15092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:13:06.423892 15092 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:13:06.425144 15092 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:13:06.425244 15092 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:06.425391 15092 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:13:06.425455 15092 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:06.457232 15092 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.3:39429
I20260925 15:13:06.457276 15219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.3:39429 every 8 connection(s)
I20260925 15:13:06.458617 15092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349162370577-12320-0/minicluster-data/ts-2/data/info.pb
I20260925 15:13:06.468487 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 15092
I20260925 15:13:06.468736 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349162370577-12320-0/minicluster-data/ts-2/wal/instance
I20260925 15:13:06.469323 12320 external_mini_cluster.cc:1428] Setting key 5bb4dca4edc6adc828f08a7e885b72f4
Sep 25 15:13:06 dist-test-slave-p468 krb5kdc[14719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349186, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.3@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
I20260925 15:13:06.478875 14955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:06.461030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.3:34597 (local address 127.12.8.62:35663)
0925 15:13:06.461281 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:06.461286 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:06.462477 (+ 1191us) server_negotiation.cc:407] Connection header received
0925 15:13:06.464350 (+ 1873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:06.464355 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:06.464461 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:06.464560 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:06.466136 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:06.466971 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:06.468481 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:06.469194 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:06.469333 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:06.473172 (+ 3839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:06.473177 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:06.473180 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:06.473215 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:06.476040 (+ 2825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:06.476771 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:06.476775 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:06.476778 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:06.476911 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:06.477308 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:06.477312 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:06.477314 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:06.477606 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:06.477725 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:06.478437 (+ 712us) server_negotiation.cc:299] Negotiation successful
0925 15:13:06.478617 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142}
I20260925 15:13:06.479773 15222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:06.461567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:35663 (local address 127.12.8.3:34597)
0925 15:13:06.462327 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:06.462368 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:06.464063 (+ 1695us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:06.464764 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:06.464771 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:06.465291 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:13:06.465925 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:06.465943 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:06.467117 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:06.467121 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:06.468328 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:06.468340 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:06.469413 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:06.469422 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:06.469633 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:06.470390 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:13:06.470425 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:13:06.472973 (+ 2548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:13:06.476189 (+ 3216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:13:06.476195 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:13:06.476209 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:13:06.476642 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:13:06.477026 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:13:06.477030 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:13:06.477032 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:13:06.477208 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:13:06.477741 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:13:06.477746 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:13:06.478053 (+ 307us) client_negotiation.cc:769] Sending connection context
0925 15:13:06.478346 (+ 293us) client_negotiation.cc:240] Negotiation successful
0925 15:13:06.478721 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":260,"threads_started":1}
I20260925 15:13:06.481146 15220 heartbeater.cc:347] Connected to a master server at 127.12.8.62:35663
I20260925 15:13:06.481581 15220 heartbeater.cc:464] Registering TS with master...
I20260925 15:13:06.482342 15220 heartbeater.cc:511] Master 127.12.8.62:35663 requested a full tablet report, sending...
I20260925 15:13:06.484017 14763 ts_manager.cc:194] Registered new tserver with Master: 5cfd3cb798ef4c2d8d872b4ff8f465ab (127.12.8.3:39429)
I20260925 15:13:06.485069 12320 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:13:06.485232 14763 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.3@KRBTEST.COM'} at 127.12.8.3:34597
W20260925 15:13:06.515043 14955 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:13:06.512881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33750 (local address 127.12.8.62:35663)
0925 15:13:06.513051 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:06.513055 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:06.513476 (+ 421us) server_negotiation.cc:407] Connection header received
0925 15:13:06.513616 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:06.513620 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:06.513689 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:06.513779 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:06.514867 (+ 1088us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33750: BlockingRecv error: recv got EOF from 127.0.0.1:33750 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 15:13:06.518292 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 14825
W20260925 15:13:06.520919 12320 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 14825: Not authorized: Client connection negotiation failed: client connection to 127.12.8.1:33193: server requires authentication, but client does not have Kerberos credentials available
I20260925 15:13:06.521101 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 14825
I20260925 15:13:06.536335 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 14959
W20260925 15:13:06.538654 12320 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 14959: Not authorized: Client connection negotiation failed: client connection to 127.12.8.2:36617: server requires authentication, but client does not have Kerberos credentials available
I20260925 15:13:06.538801 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 14959
I20260925 15:13:06.553505 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 15092
W20260925 15:13:06.556099 12320 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 15092: Not authorized: Client connection negotiation failed: client connection to 127.12.8.3:39429: server requires authentication, but client does not have Kerberos credentials available
I20260925 15:13:06.556236 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 15092
I20260925 15:13:06.571059 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 14732
I20260925 15:13:06.572121 14794 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:13:06.672463 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 14732
2026-09-25T15:13:06Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3437 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349162370577-12320-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:13:06 dist-test-slave-p468 krb5kdc[15237](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:13:06 dist-test-slave-p468 krb5kdc[15237](info): set up 2 sockets
Sep 25 15:13:06 dist-test-slave-p468 krb5kdc[15237](info): commencing operation
krb5kdc: starting...
W20260925 15:13:07.754418 12320 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349162370577-12320-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349162370577-12320-0/krb5kdc/test-user.keytab.
Sep 25 15:13:07 dist-test-slave-p468 krb5kdc[15237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349187, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:13:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:13:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.8.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab.
Entry for principal kudu/127.12.8.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab.
I20260925 15:13:07.950361 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349162370577-12320-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349162370577-12320-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349162370577-12320-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.62:42111
--webserver_interface=127.12.8.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab
--principal=kudu/127.12.8.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:37721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.62:42111
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:13:08.257937 15250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:13:08.258291 15250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:13:08.258450 15250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:13:08.265451 15250 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:13:08.265592 15250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:13:08.265704 15250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:13:08.265816 15250 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:13:08.265893 15250 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:13:08.275377 15250 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:37721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349162370577-12320-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349162370577-12320-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.8.62:42111
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab
--principal=kudu/127.12.8.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.62:42111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.8.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349162370577-12320-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:13:08.276648 15250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:13:08.277957 15250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:13:08.289630 15258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:08.289697 15255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:08.289873 15256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:13:08.290467 15250 server_base.cc:1034] running on GCE node
I20260925 15:13:08.291260 15250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:13:08.292618 15250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:13:08.293810 15250 hybrid_clock.cc:648] HybridClock initialized: now 1790349188293756 us; error 59 us; skew 500 ppm
I20260925 15:13:08.294135 15250 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:13:08 dist-test-slave-p468 krb5kdc[15237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349188, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:13:08.297183 15250 init.cc:377] Logged in from keytab as kudu/127.12.8.62@KRBTEST.COM (short username kudu)
I20260925 15:13:08.298646 15250 webserver.cc:460] Webserver started at http://127.12.8.62:40827/ using document root <none> and password file <none>
I20260925 15:13:08.299235 15250 fs_manager.cc:359] Metadata directory not provided
I20260925 15:13:08.299333 15250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:13:08.299562 15250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:13:08.302371 15250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349162370577-12320-0/minicluster-data/master-0/data/instance:
uuid: "907dcca6723a4f5084f5fbb8ece9b1ad"
format_stamp: "Formatted at 2026-09-25 15:13:08 on dist-test-slave-p468"
server_key: "a71c80cce780d3fc33e7ed6473b42f38"
server_key_iv: "fdd2f0bda324d88d563c17f78f1cff1c"
server_key_version: "encryptionkey@0"
I20260925 15:13:08.303087 15250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349162370577-12320-0/minicluster-data/master-0/wal/instance:
uuid: "907dcca6723a4f5084f5fbb8ece9b1ad"
format_stamp: "Formatted at 2026-09-25 15:13:08 on dist-test-slave-p468"
server_key: "a71c80cce780d3fc33e7ed6473b42f38"
server_key_iv: "fdd2f0bda324d88d563c17f78f1cff1c"
server_key_version: "encryptionkey@0"
I20260925 15:13:08.307885 15250 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 15:13:08.311128 15265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:08.312453 15250 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 15:13:08.312628 15250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349162370577-12320-0/minicluster-data/master-0/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349162370577-12320-0/minicluster-data/master-0/wal
uuid: "907dcca6723a4f5084f5fbb8ece9b1ad"
format_stamp: "Formatted at 2026-09-25 15:13:08 on dist-test-slave-p468"
server_key: "a71c80cce780d3fc33e7ed6473b42f38"
server_key_iv: "fdd2f0bda324d88d563c17f78f1cff1c"
server_key_version: "encryptionkey@0"
I20260925 15:13:08.312808 15250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349162370577-12320-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349162370577-12320-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349162370577-12320-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:13:08.330025 15250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:13:08.334297 15250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:13:08.334558 15250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:13:08.353184 15250 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.62:42111
I20260925 15:13:08.353228 15316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.62:42111 every 8 connection(s)
I20260925 15:13:08.354532 15250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
I20260925 15:13:08.357329 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 15250
I20260925 15:13:08.357589 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349162370577-12320-0/minicluster-data/master-0/wal/instance
I20260925 15:13:08.358493 12320 external_mini_cluster.cc:1428] Setting key 8d36aae6cdaaf9d619cdc74e599e0512
I20260925 15:13:08.359020 15317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 25 15:13:08 dist-test-slave-p468 krb5kdc[15237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349187, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
I20260925 15:13:08.372301 15317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 907dcca6723a4f5084f5fbb8ece9b1ad: Bootstrap starting.
I20260925 15:13:08.376207 15317 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 907dcca6723a4f5084f5fbb8ece9b1ad: Neither blocks nor log segments found. Creating new log.
I20260925 15:13:08.377593 15317 log.cc:824] T 00000000000000000000000000000000 P 907dcca6723a4f5084f5fbb8ece9b1ad: Log is configured to *not* fsync() on all Append() calls
I20260925 15:13:08.380203 15320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:08.360425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38702 (local address 127.12.8.62:42111)
0925 15:13:08.361192 (+ 767us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:08.361203 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:08.361240 (+ 37us) server_negotiation.cc:407] Connection header received
0925 15:13:08.362812 (+ 1572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:08.362832 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:08.363198 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:08.363580 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:08.365098 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:08.366507 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:08.367754 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:08.368410 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:08.368604 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:08.373003 (+ 4399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:08.373011 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:08.373025 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:08.373061 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:08.376030 (+ 2969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:08.376895 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:08.376901 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:08.376906 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:08.377063 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:08.377509 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:08.377512 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:08.377513 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:08.378064 (+ 551us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:08.378398 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:08.378829 (+ 431us) server_negotiation.cc:299] Negotiation successful
0925 15:13:08.379213 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":363,"threads_started":1}
I20260925 15:13:08.380801 15317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 907dcca6723a4f5084f5fbb8ece9b1ad: No bootstrap required, opened a new log
I20260925 15:13:08.383813 12320 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:13:08.388945 15317 raft_consensus.cc:348] T 00000000000000000000000000000000 P 907dcca6723a4f5084f5fbb8ece9b1ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "907dcca6723a4f5084f5fbb8ece9b1ad" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 42111 } }
I20260925 15:13:08.389431 15317 raft_consensus.cc:374] T 00000000000000000000000000000000 P 907dcca6723a4f5084f5fbb8ece9b1ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:13:08.389560 15317 raft_consensus.cc:729] T 00000000000000000000000000000000 P 907dcca6723a4f5084f5fbb8ece9b1ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 907dcca6723a4f5084f5fbb8ece9b1ad, State: Initialized, Role: FOLLOWER
I20260925 15:13:08.390300 15317 consensus_queue.cc:260] T 00000000000000000000000000000000 P 907dcca6723a4f5084f5fbb8ece9b1ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "907dcca6723a4f5084f5fbb8ece9b1ad" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 42111 } }
I20260925 15:13:08.390628 15317 raft_consensus.cc:388] T 00000000000000000000000000000000 P 907dcca6723a4f5084f5fbb8ece9b1ad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:13:08.390774 15317 raft_consensus.cc:482] T 00000000000000000000000000000000 P 907dcca6723a4f5084f5fbb8ece9b1ad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:13:08.390944 15317 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 907dcca6723a4f5084f5fbb8ece9b1ad [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:13:08.394901 15317 raft_consensus.cc:504] T 00000000000000000000000000000000 P 907dcca6723a4f5084f5fbb8ece9b1ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "907dcca6723a4f5084f5fbb8ece9b1ad" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 42111 } }
I20260925 15:13:08.395524 15317 leader_election.cc:302] T 00000000000000000000000000000000 P 907dcca6723a4f5084f5fbb8ece9b1ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 907dcca6723a4f5084f5fbb8ece9b1ad; no voters:
I20260925 15:13:08.396185 15317 leader_election.cc:288] T 00000000000000000000000000000000 P 907dcca6723a4f5084f5fbb8ece9b1ad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:13:08.396306 15323 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 907dcca6723a4f5084f5fbb8ece9b1ad [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:13:08.397876 15323 raft_consensus.cc:686] T 00000000000000000000000000000000 P 907dcca6723a4f5084f5fbb8ece9b1ad [term 1 LEADER]: Becoming Leader. State: Replica: 907dcca6723a4f5084f5fbb8ece9b1ad, State: Running, Role: LEADER
I20260925 15:13:08.398466 15323 consensus_queue.cc:237] T 00000000000000000000000000000000 P 907dcca6723a4f5084f5fbb8ece9b1ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "907dcca6723a4f5084f5fbb8ece9b1ad" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 42111 } }
I20260925 15:13:08.398765 15317 sys_catalog.cc:564] T 00000000000000000000000000000000 P 907dcca6723a4f5084f5fbb8ece9b1ad [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:13:08.401077 15325 sys_catalog.cc:455] T 00000000000000000000000000000000 P 907dcca6723a4f5084f5fbb8ece9b1ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 907dcca6723a4f5084f5fbb8ece9b1ad. Latest consensus state: current_term: 1 leader_uuid: "907dcca6723a4f5084f5fbb8ece9b1ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "907dcca6723a4f5084f5fbb8ece9b1ad" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 42111 } } }
I20260925 15:13:08.401480 15325 sys_catalog.cc:458] T 00000000000000000000000000000000 P 907dcca6723a4f5084f5fbb8ece9b1ad [sys.catalog]: This master's current role is: LEADER
I20260925 15:13:08.413154 15328 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:13:08.415809 15324 sys_catalog.cc:455] T 00000000000000000000000000000000 P 907dcca6723a4f5084f5fbb8ece9b1ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "907dcca6723a4f5084f5fbb8ece9b1ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "907dcca6723a4f5084f5fbb8ece9b1ad" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 42111 } } }
I20260925 15:13:08.416162 15324 sys_catalog.cc:458] T 00000000000000000000000000000000 P 907dcca6723a4f5084f5fbb8ece9b1ad [sys.catalog]: This master's current role is: LEADER
I20260925 15:13:08.422786 15328 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:13:08.434813 15320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:08.425657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38714 (local address 127.12.8.62:42111)
0925 15:13:08.425911 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:08.425920 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:08.426642 (+ 722us) server_negotiation.cc:407] Connection header received
0925 15:13:08.426858 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:08.426868 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:08.426947 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:08.427044 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:08.428423 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:08.429824 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:08.431132 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:08.431552 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:08.431697 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:08.432798 (+ 1101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:08.432808 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:08.432813 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:13:08.433757 (+ 944us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:08.433802 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:13:08.433835 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:08.433925 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:08.434162 (+ 237us) server_negotiation.cc:299] Negotiation successful
0925 15:13:08.434319 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260925 15:13:08.435458 15328 catalog_manager.cc:1347] Generated new cluster ID: ef443ee4034d43f08d4efa895a714302
I20260925 15:13:08.435591 15328 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:13:08.449446 15328 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:13:08.450747 15328 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:13:08.458881 15328 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 907dcca6723a4f5084f5fbb8ece9b1ad: Generated new TSK 0
I20260925 15:13:08.459637 15328 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:13:08.470649 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 15250
I20260925 15:13:08.471699 15312 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:13:08.561108 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 15250
2026-09-25T15:13:08Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1882 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349162370577-12320-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:13:08 dist-test-slave-p468 krb5kdc[15351](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:13:08 dist-test-slave-p468 krb5kdc[15351](info): set up 2 sockets
Sep 25 15:13:08 dist-test-slave-p468 krb5kdc[15351](info): commencing operation
krb5kdc: starting...
W20260925 15:13:09.658555 12320 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349162370577-12320-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349162370577-12320-0/krb5kdc/test-user.keytab.
Sep 25 15:13:09 dist-test-slave-p468 krb5kdc[15351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349189, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:13:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:13:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.8.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab.
Entry for principal kudu/127.12.8.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab.
I20260925 15:13:09.845641 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349162370577-12320-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349162370577-12320-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349162370577-12320-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.62:34581
--webserver_interface=127.12.8.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab
--principal=kudu/127.12.8.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:44351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.62:34581
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:13:10.156221 15364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:13:10.156478 15364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:13:10.156642 15364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:13:10.163733 15364 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:13:10.163826 15364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:13:10.163934 15364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:13:10.164033 15364 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:13:10.164120 15364 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:13:10.173580 15364 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:44351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349162370577-12320-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349162370577-12320-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.8.62:34581
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab
--principal=kudu/127.12.8.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.62:34581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.8.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349162370577-12320-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:13:10.174830 15364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:13:10.176095 15364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:13:10.187798 15370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:10.187840 15372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:10.187883 15369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:13:10.188793 15364 server_base.cc:1034] running on GCE node
I20260925 15:13:10.189597 15364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:13:10.191004 15364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:13:10.192239 15364 hybrid_clock.cc:648] HybridClock initialized: now 1790349190192197 us; error 39 us; skew 500 ppm
I20260925 15:13:10.192577 15364 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:13:10 dist-test-slave-p468 krb5kdc[15351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349190, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:13:10.200516 15364 init.cc:377] Logged in from keytab as kudu/127.12.8.62@KRBTEST.COM (short username kudu)
I20260925 15:13:10.201880 15364 webserver.cc:460] Webserver started at http://127.12.8.62:43333/ using document root <none> and password file <none>
I20260925 15:13:10.202591 15364 fs_manager.cc:359] Metadata directory not provided
I20260925 15:13:10.202728 15364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:13:10.203025 15364 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:13:10.205909 15364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349162370577-12320-0/minicluster-data/master-0/data/instance:
uuid: "9d2cc5db00dd42608d284de54f9f33bc"
format_stamp: "Formatted at 2026-09-25 15:13:10 on dist-test-slave-p468"
server_key: "f31c57f5012e3a530947122f4af0d911"
server_key_iv: "540f2cb4ccec5e46278c6ad69246b35b"
server_key_version: "encryptionkey@0"
I20260925 15:13:10.206719 15364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349162370577-12320-0/minicluster-data/master-0/wal/instance:
uuid: "9d2cc5db00dd42608d284de54f9f33bc"
format_stamp: "Formatted at 2026-09-25 15:13:10 on dist-test-slave-p468"
server_key: "f31c57f5012e3a530947122f4af0d911"
server_key_iv: "540f2cb4ccec5e46278c6ad69246b35b"
server_key_version: "encryptionkey@0"
I20260925 15:13:10.211560 15364 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 15:13:10.214720 15379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:10.215976 15364 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 15:13:10.216146 15364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349162370577-12320-0/minicluster-data/master-0/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349162370577-12320-0/minicluster-data/master-0/wal
uuid: "9d2cc5db00dd42608d284de54f9f33bc"
format_stamp: "Formatted at 2026-09-25 15:13:10 on dist-test-slave-p468"
server_key: "f31c57f5012e3a530947122f4af0d911"
server_key_iv: "540f2cb4ccec5e46278c6ad69246b35b"
server_key_version: "encryptionkey@0"
I20260925 15:13:10.216333 15364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349162370577-12320-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349162370577-12320-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349162370577-12320-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:13:10.245514 15364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:13:10.249392 15364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:13:10.249689 15364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:13:10.267514 15364 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.62:34581
I20260925 15:13:10.267545 15430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.62:34581 every 8 connection(s)
I20260925 15:13:10.268813 15364 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
I20260925 15:13:10.271546 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 15364
I20260925 15:13:10.271764 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349162370577-12320-0/minicluster-data/master-0/wal/instance
I20260925 15:13:10.272359 15431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:13:10.272436 12320 external_mini_cluster.cc:1428] Setting key d9367ddf2b041079236d380560daf33b
I20260925 15:13:10.286259 15431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d2cc5db00dd42608d284de54f9f33bc: Bootstrap starting.
Sep 25 15:13:10 dist-test-slave-p468 krb5kdc[15351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349189, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
I20260925 15:13:10.289835 15431 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d2cc5db00dd42608d284de54f9f33bc: Neither blocks nor log segments found. Creating new log.
I20260925 15:13:10.291133 15431 log.cc:824] T 00000000000000000000000000000000 P 9d2cc5db00dd42608d284de54f9f33bc: Log is configured to *not* fsync() on all Append() calls
I20260925 15:13:10.294442 15431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d2cc5db00dd42608d284de54f9f33bc: No bootstrap required, opened a new log
I20260925 15:13:10.295742 15434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:10.274358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50956 (local address 127.12.8.62:34581)
0925 15:13:10.275134 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:10.275145 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:10.275185 (+ 40us) server_negotiation.cc:407] Connection header received
0925 15:13:10.276836 (+ 1651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:10.276856 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:10.277225 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:10.277535 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:10.279238 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:10.280465 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:10.282270 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:10.282955 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:10.283104 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:10.287706 (+ 4602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:10.287717 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:10.287737 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:10.287789 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:10.291485 (+ 3696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:10.292303 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:10.292308 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:10.292313 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:10.292494 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:10.292852 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:10.292856 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:10.292859 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:10.293482 (+ 623us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:10.293790 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:10.294091 (+ 301us) server_negotiation.cc:299] Negotiation successful
0925 15:13:10.294514 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":365,"threads_started":1}
I20260925 15:13:10.302646 15431 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d2cc5db00dd42608d284de54f9f33bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d2cc5db00dd42608d284de54f9f33bc" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 34581 } }
I20260925 15:13:10.302973 15431 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d2cc5db00dd42608d284de54f9f33bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:13:10.303050 15431 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d2cc5db00dd42608d284de54f9f33bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d2cc5db00dd42608d284de54f9f33bc, State: Initialized, Role: FOLLOWER
I20260925 15:13:10.303527 15431 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d2cc5db00dd42608d284de54f9f33bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d2cc5db00dd42608d284de54f9f33bc" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 34581 } }
I20260925 15:13:10.303732 15431 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d2cc5db00dd42608d284de54f9f33bc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:13:10.303895 15431 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d2cc5db00dd42608d284de54f9f33bc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:13:10.304059 15431 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d2cc5db00dd42608d284de54f9f33bc [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:13:10.306143 15431 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d2cc5db00dd42608d284de54f9f33bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d2cc5db00dd42608d284de54f9f33bc" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 34581 } }
I20260925 15:13:10.306712 15431 leader_election.cc:302] T 00000000000000000000000000000000 P 9d2cc5db00dd42608d284de54f9f33bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d2cc5db00dd42608d284de54f9f33bc; no voters:
I20260925 15:13:10.307255 15431 leader_election.cc:288] T 00000000000000000000000000000000 P 9d2cc5db00dd42608d284de54f9f33bc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:13:10.307389 15436 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d2cc5db00dd42608d284de54f9f33bc [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:13:10.308110 15436 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d2cc5db00dd42608d284de54f9f33bc [term 1 LEADER]: Becoming Leader. State: Replica: 9d2cc5db00dd42608d284de54f9f33bc, State: Running, Role: LEADER
I20260925 15:13:10.308584 15436 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d2cc5db00dd42608d284de54f9f33bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d2cc5db00dd42608d284de54f9f33bc" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 34581 } }
I20260925 15:13:10.309412 15431 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d2cc5db00dd42608d284de54f9f33bc [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:13:10.311553 15437 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d2cc5db00dd42608d284de54f9f33bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9d2cc5db00dd42608d284de54f9f33bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d2cc5db00dd42608d284de54f9f33bc" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 34581 } } }
I20260925 15:13:10.311960 15437 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d2cc5db00dd42608d284de54f9f33bc [sys.catalog]: This master's current role is: LEADER
I20260925 15:13:10.311544 15438 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d2cc5db00dd42608d284de54f9f33bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d2cc5db00dd42608d284de54f9f33bc. Latest consensus state: current_term: 1 leader_uuid: "9d2cc5db00dd42608d284de54f9f33bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d2cc5db00dd42608d284de54f9f33bc" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 34581 } } }
I20260925 15:13:10.312096 15438 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d2cc5db00dd42608d284de54f9f33bc [sys.catalog]: This master's current role is: LEADER
I20260925 15:13:10.314147 15442 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:13:10.320252 15442 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:13:10.329730 15442 catalog_manager.cc:1347] Generated new cluster ID: d361f857ba38402cafbe680a7af05771
I20260925 15:13:10.329843 15442 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:13:10.339723 15442 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:13:10.341007 15442 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:13:10.355958 15442 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d2cc5db00dd42608d284de54f9f33bc: Generated new TSK 0
I20260925 15:13:10.356786 15442 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.8.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.1.keytab.
Entry for principal kudu/127.12.8.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.1.keytab.
I20260925 15:13:10.409569 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349162370577-12320-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349162370577-12320-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349162370577-12320-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349162370577-12320-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.1:0
--local_ip_for_outbound_sockets=127.12.8.1
--webserver_interface=127.12.8.1
--webserver_port=0
--tserver_master_addrs=127.12.8.62:34581
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.1.keytab
--principal=kudu/127.12.8.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:44351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:13:10.720636 15457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:13:10.720993 15457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:13:10.721168 15457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:13:10.728237 15457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:13:10.728382 15457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:13:10.728603 15457 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.1
I20260925 15:13:10.737963 15457 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:44351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349162370577-12320-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349162370577-12320-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.1.keytab
--principal=kudu/127.12.8.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349162370577-12320-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.8.1
--webserver_port=0
--tserver_master_addrs=127.12.8.62:34581
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.1
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349162370577-12320-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:13:10.739207 15457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:13:10.740432 15457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:13:10.752848 15462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:10.752951 15465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:10.752919 15463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:13:10.754323 15457 server_base.cc:1034] running on GCE node
I20260925 15:13:10.754868 15457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:13:10.755744 15457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:13:10.756969 15457 hybrid_clock.cc:648] HybridClock initialized: now 1790349190756934 us; error 49 us; skew 500 ppm
I20260925 15:13:10.757266 15457 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:13:10 dist-test-slave-p468 krb5kdc[15351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349190, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:13:10.760253 15457 init.cc:377] Logged in from keytab as kudu/127.12.8.1@KRBTEST.COM (short username kudu)
I20260925 15:13:10.761562 15457 webserver.cc:460] Webserver started at http://127.12.8.1:46611/ using document root <none> and password file <none>
I20260925 15:13:10.762270 15457 fs_manager.cc:359] Metadata directory not provided
I20260925 15:13:10.762416 15457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:13:10.762706 15457 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:13:10.765403 15457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349162370577-12320-0/minicluster-data/ts-0/data/instance:
uuid: "bd696eb5dda94b0ca8e5a72c078067d4"
format_stamp: "Formatted at 2026-09-25 15:13:10 on dist-test-slave-p468"
server_key: "22d4f494c4cd4e620b5c8610b8946129"
server_key_iv: "993f522afc2faa4563c47e3ae821164d"
server_key_version: "encryptionkey@0"
I20260925 15:13:10.766168 15457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349162370577-12320-0/minicluster-data/ts-0/wal/instance:
uuid: "bd696eb5dda94b0ca8e5a72c078067d4"
format_stamp: "Formatted at 2026-09-25 15:13:10 on dist-test-slave-p468"
server_key: "22d4f494c4cd4e620b5c8610b8946129"
server_key_iv: "993f522afc2faa4563c47e3ae821164d"
server_key_version: "encryptionkey@0"
I20260925 15:13:10.770919 15457 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260925 15:13:10.774585 15472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:10.775831 15457 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260925 15:13:10.776006 15457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349162370577-12320-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349162370577-12320-0/minicluster-data/ts-0/wal
uuid: "bd696eb5dda94b0ca8e5a72c078067d4"
format_stamp: "Formatted at 2026-09-25 15:13:10 on dist-test-slave-p468"
server_key: "22d4f494c4cd4e620b5c8610b8946129"
server_key_iv: "993f522afc2faa4563c47e3ae821164d"
server_key_version: "encryptionkey@0"
I20260925 15:13:10.776221 15457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349162370577-12320-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349162370577-12320-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349162370577-12320-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:13:10.795305 15457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:13:10.799253 15457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:13:10.799538 15457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:13:10.800437 15457 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:13:10.801705 15457 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:13:10.801807 15457 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:10.801949 15457 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:13:10.802026 15457 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:10.834825 15457 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.1:41983
I20260925 15:13:10.834879 15584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.1:41983 every 8 connection(s)
I20260925 15:13:10.836066 15457 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349162370577-12320-0/minicluster-data/ts-0/data/info.pb
I20260925 15:13:10.838465 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 15457
I20260925 15:13:10.838816 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349162370577-12320-0/minicluster-data/ts-0/wal/instance
I20260925 15:13:10.839613 12320 external_mini_cluster.cc:1428] Setting key 08fedebeeee764482176ac3a92be4b03
Sep 25 15:13:10 dist-test-slave-p468 krb5kdc[15351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349190, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.1@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.8.2@KRBTEST.COM; defaulting to no policy
I20260925 15:13:10.856436 15587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:10.838798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.1:49795 (local address 127.12.8.62:34581)
0925 15:13:10.839221 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:10.839225 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:10.840376 (+ 1151us) server_negotiation.cc:407] Connection header received
0925 15:13:10.842615 (+ 2239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:10.842620 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:10.842706 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:10.842819 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:10.844262 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:10.844973 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:10.846566 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:10.846972 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:10.847112 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:10.850710 (+ 3598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:10.850715 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:10.850718 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:10.850748 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:10.853308 (+ 2560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:10.854015 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:10.854024 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:10.854027 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:10.854135 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:10.854589 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:10.854594 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:10.854597 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:10.854878 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:10.855078 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:10.855838 (+ 760us) server_negotiation.cc:299] Negotiation successful
0925 15:13:10.856043 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":238,"threads_started":1}
I20260925 15:13:10.857470 15588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:10.839361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:34581 (local address 127.12.8.1:49795)
0925 15:13:10.840172 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:10.840213 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:10.842348 (+ 2135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:10.843051 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:10.843060 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:10.843559 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:13:10.844092 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:10.844108 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:10.845112 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:10.845118 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:10.846421 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:10.846433 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:10.847193 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:10.847203 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:10.847438 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:10.848115 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:13:10.848136 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:13:10.850501 (+ 2365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:13:10.853448 (+ 2947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:13:10.853457 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:13:10.853471 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:13:10.853886 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:13:10.854280 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:13:10.854285 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:13:10.854289 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:13:10.854469 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:13:10.855053 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:13:10.855061 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:13:10.855371 (+ 310us) client_negotiation.cc:769] Sending connection context
0925 15:13:10.855745 (+ 374us) client_negotiation.cc:240] Negotiation successful
0925 15:13:10.856245 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":370,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.2@KRBTEST.COM" created.
I20260925 15:13:10.859071 15585 heartbeater.cc:347] Connected to a master server at 127.12.8.62:34581
I20260925 15:13:10.859378 15585 heartbeater.cc:464] Registering TS with master...
I20260925 15:13:10.860095 15585 heartbeater.cc:511] Master 127.12.8.62:34581 requested a full tablet report, sending...
I20260925 15:13:10.862056 15396 ts_manager.cc:194] Registered new tserver with Master: bd696eb5dda94b0ca8e5a72c078067d4 (127.12.8.1:41983)
I20260925 15:13:10.863474 15396 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.1@KRBTEST.COM'} at 127.12.8.1:49795
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.2.keytab.
Entry for principal kudu/127.12.8.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.2.keytab.
I20260925 15:13:10.881136 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349162370577-12320-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349162370577-12320-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349162370577-12320-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349162370577-12320-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.2:0
--local_ip_for_outbound_sockets=127.12.8.2
--webserver_interface=127.12.8.2
--webserver_port=0
--tserver_master_addrs=127.12.8.62:34581
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.2.keytab
--principal=kudu/127.12.8.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:44351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:13:11.198506 15591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:13:11.198771 15591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:13:11.198863 15591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:13:11.205816 15591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:13:11.205920 15591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:13:11.206028 15591 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.2
I20260925 15:13:11.215651 15591 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:44351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349162370577-12320-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349162370577-12320-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.2.keytab
--principal=kudu/127.12.8.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349162370577-12320-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.8.2
--webserver_port=0
--tserver_master_addrs=127.12.8.62:34581
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.2
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349162370577-12320-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:13:11.216900 15591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:13:11.218134 15591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:13:11.231137 15596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:11.231267 15597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:11.231207 15599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:13:11.231868 15591 server_base.cc:1034] running on GCE node
I20260925 15:13:11.232456 15591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:13:11.233379 15591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:13:11.234606 15591 hybrid_clock.cc:648] HybridClock initialized: now 1790349191234560 us; error 52 us; skew 500 ppm
I20260925 15:13:11.234910 15591 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:13:11 dist-test-slave-p468 krb5kdc[15351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349191, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:13:11.238083 15591 init.cc:377] Logged in from keytab as kudu/127.12.8.2@KRBTEST.COM (short username kudu)
I20260925 15:13:11.239710 15591 webserver.cc:460] Webserver started at http://127.12.8.2:34083/ using document root <none> and password file <none>
I20260925 15:13:11.240432 15591 fs_manager.cc:359] Metadata directory not provided
I20260925 15:13:11.240584 15591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:13:11.240904 15591 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:13:11.243986 15591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349162370577-12320-0/minicluster-data/ts-1/data/instance:
uuid: "dcc433d982f048b290f7ef44fd30caf1"
format_stamp: "Formatted at 2026-09-25 15:13:11 on dist-test-slave-p468"
server_key: "481505029df4d9c79631a0bea5b8d120"
server_key_iv: "4197da79a46e6e6775f4b5d01764d35c"
server_key_version: "encryptionkey@0"
I20260925 15:13:11.244813 15591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349162370577-12320-0/minicluster-data/ts-1/wal/instance:
uuid: "dcc433d982f048b290f7ef44fd30caf1"
format_stamp: "Formatted at 2026-09-25 15:13:11 on dist-test-slave-p468"
server_key: "481505029df4d9c79631a0bea5b8d120"
server_key_iv: "4197da79a46e6e6775f4b5d01764d35c"
server_key_version: "encryptionkey@0"
I20260925 15:13:11.250128 15591 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 15:13:11.253531 15606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:11.254908 15591 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:13:11.255043 15591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349162370577-12320-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349162370577-12320-0/minicluster-data/ts-1/wal
uuid: "dcc433d982f048b290f7ef44fd30caf1"
format_stamp: "Formatted at 2026-09-25 15:13:11 on dist-test-slave-p468"
server_key: "481505029df4d9c79631a0bea5b8d120"
server_key_iv: "4197da79a46e6e6775f4b5d01764d35c"
server_key_version: "encryptionkey@0"
I20260925 15:13:11.255211 15591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349162370577-12320-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349162370577-12320-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349162370577-12320-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:13:11.275039 15591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:13:11.279699 15591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:13:11.279999 15591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:13:11.280999 15591 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:13:11.282357 15591 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:13:11.282473 15591 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:11.282632 15591 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:13:11.282711 15591 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:11.317193 15591 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.2:34025
I20260925 15:13:11.317236 15718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.2:34025 every 8 connection(s)
I20260925 15:13:11.318655 15591 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349162370577-12320-0/minicluster-data/ts-1/data/info.pb
I20260925 15:13:11.328554 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 15591
I20260925 15:13:11.328740 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349162370577-12320-0/minicluster-data/ts-1/wal/instance
I20260925 15:13:11.329249 12320 external_mini_cluster.cc:1428] Setting key 623f2f28b7def3edbc1b8a948f92fb0a
Sep 25 15:13:11 dist-test-slave-p468 krb5kdc[15351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349191, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.2@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
I20260925 15:13:11.338168 15587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:11.321357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.2:33483 (local address 127.12.8.62:34581)
0925 15:13:11.321542 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:11.321548 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:11.322771 (+ 1223us) server_negotiation.cc:407] Connection header received
0925 15:13:11.324481 (+ 1710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:11.324487 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:11.324603 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:11.324713 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:11.326207 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:11.326908 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:11.328405 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:11.328929 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:11.329067 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:11.332794 (+ 3727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:11.332804 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:11.332807 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:11.332835 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:11.335382 (+ 2547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:11.336043 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:11.336049 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:11.336054 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:11.336185 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:11.336596 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:11.336603 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:11.336606 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:11.336838 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:11.336983 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:11.337621 (+ 638us) server_negotiation.cc:299] Negotiation successful
0925 15:13:11.337841 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 15:13:11.339107 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:11.321882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:34581 (local address 127.12.8.2:33483)
0925 15:13:11.322619 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:11.322660 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:11.324220 (+ 1560us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:11.324942 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:11.324950 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:11.325440 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:13:11.326007 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:11.326024 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:11.327058 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:11.327069 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:11.328266 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:11.328277 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:11.329162 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:11.329169 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:11.329432 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:11.330148 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:13:11.330222 (+ 74us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:13:11.332603 (+ 2381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:13:11.335512 (+ 2909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:13:11.335520 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:13:11.335535 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:13:11.335929 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:13:11.336301 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:13:11.336307 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:13:11.336310 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:13:11.336484 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:13:11.336957 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:13:11.336965 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:13:11.337276 (+ 311us) client_negotiation.cc:769] Sending connection context
0925 15:13:11.337527 (+ 251us) client_negotiation.cc:240] Negotiation successful
0925 15:13:11.338010 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":330,"threads_started":1}
I20260925 15:13:11.340490 15719 heartbeater.cc:347] Connected to a master server at 127.12.8.62:34581
I20260925 15:13:11.340807 15719 heartbeater.cc:464] Registering TS with master...
I20260925 15:13:11.341707 15719 heartbeater.cc:511] Master 127.12.8.62:34581 requested a full tablet report, sending...
I20260925 15:13:11.343402 15396 ts_manager.cc:194] Registered new tserver with Master: dcc433d982f048b290f7ef44fd30caf1 (127.12.8.2:34025)
I20260925 15:13:11.344744 15396 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.2@KRBTEST.COM'} at 127.12.8.2:33483
WARNING: no policy specified for kudu/127.12.8.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.3.keytab.
Entry for principal kudu/127.12.8.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.3.keytab.
I20260925 15:13:11.375459 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349162370577-12320-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349162370577-12320-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349162370577-12320-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349162370577-12320-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.3:0
--local_ip_for_outbound_sockets=127.12.8.3
--webserver_interface=127.12.8.3
--webserver_port=0
--tserver_master_addrs=127.12.8.62:34581
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.3.keytab
--principal=kudu/127.12.8.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:44351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:13:11.686241 15724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:13:11.686514 15724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:13:11.686674 15724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:13:11.693869 15724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:13:11.693971 15724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:13:11.694123 15724 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.3
I20260925 15:13:11.703430 15724 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:44351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349162370577-12320-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349162370577-12320-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.3.keytab
--principal=kudu/127.12.8.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349162370577-12320-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.8.3
--webserver_port=0
--tserver_master_addrs=127.12.8.62:34581
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.3
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349162370577-12320-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:13:11.704674 15724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:13:11.705932 15724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:13:11.718870 15729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:11.718915 15730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:11.718936 15732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:13:11.719705 15724 server_base.cc:1034] running on GCE node
I20260925 15:13:11.720283 15724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:13:11.721244 15724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:13:11.722437 15724 hybrid_clock.cc:648] HybridClock initialized: now 1790349191722397 us; error 27 us; skew 500 ppm
I20260925 15:13:11.722712 15724 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:13:11 dist-test-slave-p468 krb5kdc[15351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349191, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:13:11.725771 15724 init.cc:377] Logged in from keytab as kudu/127.12.8.3@KRBTEST.COM (short username kudu)
I20260925 15:13:11.727169 15724 webserver.cc:460] Webserver started at http://127.12.8.3:34001/ using document root <none> and password file <none>
I20260925 15:13:11.727869 15724 fs_manager.cc:359] Metadata directory not provided
I20260925 15:13:11.728021 15724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:13:11.728329 15724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:13:11.731366 15724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349162370577-12320-0/minicluster-data/ts-2/data/instance:
uuid: "8538fa2d153e476a9d013d845637269d"
format_stamp: "Formatted at 2026-09-25 15:13:11 on dist-test-slave-p468"
server_key: "b88f3dc8a8322c73b4a55a5ef89ccb8b"
server_key_iv: "d0b35ef1164d1f2fb7be9a783a985ac4"
server_key_version: "encryptionkey@0"
I20260925 15:13:11.732167 15724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349162370577-12320-0/minicluster-data/ts-2/wal/instance:
uuid: "8538fa2d153e476a9d013d845637269d"
format_stamp: "Formatted at 2026-09-25 15:13:11 on dist-test-slave-p468"
server_key: "b88f3dc8a8322c73b4a55a5ef89ccb8b"
server_key_iv: "d0b35ef1164d1f2fb7be9a783a985ac4"
server_key_version: "encryptionkey@0"
I20260925 15:13:11.737131 15724 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260925 15:13:11.740353 15739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:11.741627 15724 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260925 15:13:11.741797 15724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349162370577-12320-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349162370577-12320-0/minicluster-data/ts-2/wal
uuid: "8538fa2d153e476a9d013d845637269d"
format_stamp: "Formatted at 2026-09-25 15:13:11 on dist-test-slave-p468"
server_key: "b88f3dc8a8322c73b4a55a5ef89ccb8b"
server_key_iv: "d0b35ef1164d1f2fb7be9a783a985ac4"
server_key_version: "encryptionkey@0"
I20260925 15:13:11.741991 15724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349162370577-12320-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349162370577-12320-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349162370577-12320-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:13:11.760908 15724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:13:11.765010 15724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:13:11.765296 15724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:13:11.766232 15724 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:13:11.767511 15724 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:13:11.767624 15724 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:11.767782 15724 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:13:11.767855 15724 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:11.800097 15724 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.3:34227
I20260925 15:13:11.800148 15851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.3:34227 every 8 connection(s)
I20260925 15:13:11.801326 15724 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349162370577-12320-0/minicluster-data/ts-2/data/info.pb
I20260925 15:13:11.801928 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 15724
I20260925 15:13:11.802122 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349162370577-12320-0/minicluster-data/ts-2/wal/instance
I20260925 15:13:11.802743 12320 external_mini_cluster.cc:1428] Setting key 92a517e2821806599e8f7074d2b6e1a1
Sep 25 15:13:11 dist-test-slave-p468 krb5kdc[15351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349191, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.3@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
I20260925 15:13:11.821488 15587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:11.804186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.3:49253 (local address 127.12.8.62:34581)
0925 15:13:11.804400 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:11.804404 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:11.805702 (+ 1298us) server_negotiation.cc:407] Connection header received
0925 15:13:11.807542 (+ 1840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:11.807546 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:11.807607 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:11.807694 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:11.809032 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:11.809842 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:11.811501 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:11.811913 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:11.812045 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:11.815382 (+ 3337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:11.815388 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:11.815393 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:11.815434 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:11.818521 (+ 3087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:11.819211 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:11.819218 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:11.819221 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:11.819345 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:11.819777 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:11.819782 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:11.819786 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:11.820041 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:11.820198 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:11.820952 (+ 754us) server_negotiation.cc:299] Negotiation successful
0925 15:13:11.821159 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 15:13:11.822438 15854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:11.804733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:34581 (local address 127.12.8.3:49253)
0925 15:13:11.805528 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:11.805587 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:11.807226 (+ 1639us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:11.807884 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:11.807891 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:11.808362 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:13:11.808895 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:11.808910 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:11.810002 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:11.810007 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:11.811361 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:11.811372 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:11.812133 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:11.812143 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:11.812388 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:11.813062 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:13:11.813090 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:13:11.815172 (+ 2082us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:13:11.818656 (+ 3484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:13:11.818665 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:13:11.818680 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:13:11.819084 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:13:11.819463 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:13:11.819468 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:13:11.819472 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:13:11.819658 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:13:11.820176 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:13:11.820183 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:13:11.820585 (+ 402us) client_negotiation.cc:769] Sending connection context
0925 15:13:11.820840 (+ 255us) client_negotiation.cc:240] Negotiation successful
0925 15:13:11.821321 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":324,"threads_started":1}
I20260925 15:13:11.823747 15852 heartbeater.cc:347] Connected to a master server at 127.12.8.62:34581
I20260925 15:13:11.824062 15852 heartbeater.cc:464] Registering TS with master...
I20260925 15:13:11.824753 15852 heartbeater.cc:511] Master 127.12.8.62:34581 requested a full tablet report, sending...
I20260925 15:13:11.826093 15396 ts_manager.cc:194] Registered new tserver with Master: 8538fa2d153e476a9d013d845637269d (127.12.8.3:34227)
I20260925 15:13:11.826879 15396 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.3@KRBTEST.COM'} at 127.12.8.3:49253
I20260925 15:13:11.830299 12320 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 15:13:11 dist-test-slave-p468 krb5kdc[15351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349191, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
I20260925 15:13:11.866145 15585 heartbeater.cc:503] Master 127.12.8.62:34581 was elected leader, sending a full tablet report...
Sep 25 15:13:11 dist-test-slave-p468 krb5kdc[15351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349191, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
I20260925 15:13:11.880709 15587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:11.867509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50966 (local address 127.12.8.62:34581)
0925 15:13:11.867785 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:11.867807 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:11.868322 (+ 515us) server_negotiation.cc:407] Connection header received
0925 15:13:11.868461 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:11.868464 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:11.868522 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:11.868599 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:11.869816 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:11.870587 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:11.871769 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:11.872150 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:11.872237 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:11.875909 (+ 3672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:11.875914 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:11.875917 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:11.875949 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:11.878334 (+ 2385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:11.878885 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:11.878889 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:11.878891 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:11.879003 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:11.879375 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:11.879379 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:11.879381 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:11.879757 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:11.879953 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:11.880279 (+ 326us) server_negotiation.cc:299] Negotiation successful
0925 15:13:11.880492 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
W20260925 15:13:11.881203 15396 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:50966
I20260925 15:13:11.882781 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 15457
Sep 25 15:13:11 dist-test-slave-p468 krb5kdc[15351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349191, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.12.8.1@KRBTEST.COM
I20260925 15:13:11.898037 15863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:11.883835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35272 (local address 127.12.8.1:41983)
0925 15:13:11.884397 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:11.884405 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:11.884427 (+ 22us) server_negotiation.cc:407] Connection header received
0925 15:13:11.884479 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:11.884485 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:11.884698 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:11.884826 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:11.885986 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:11.887099 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:11.888356 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:11.888938 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:11.889037 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:11.892935 (+ 3898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:11.892946 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:11.892962 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:11.893001 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:11.895687 (+ 2686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:11.896366 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:11.896373 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:11.896375 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:11.896496 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:11.896820 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:11.896823 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:11.896825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:11.897135 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:11.897341 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:11.897572 (+ 231us) server_negotiation.cc:299] Negotiation successful
0925 15:13:11.897753 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":279,"threads_started":1}
W20260925 15:13:11.899261 15580 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:35272
W20260925 15:13:11.900291 12320 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 15457: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 15:13:11.900501 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 15457
I20260925 15:13:11.915943 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 15591
Sep 25 15:13:11 dist-test-slave-p468 krb5kdc[15351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349191, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.12.8.2@KRBTEST.COM
I20260925 15:13:11.931792 15864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:11.916959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54734 (local address 127.12.8.2:34025)
0925 15:13:11.917563 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:11.917570 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:11.917594 (+ 24us) server_negotiation.cc:407] Connection header received
0925 15:13:11.917654 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:11.917660 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:11.917851 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:11.918036 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:11.919464 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:11.920589 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:11.921836 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:11.922446 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:11.922578 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:11.926449 (+ 3871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:11.926459 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:11.926473 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:11.926505 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:11.929050 (+ 2545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:11.929691 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:11.929703 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:11.929706 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:11.929824 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:11.930176 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:11.930182 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:11.930253 (+ 71us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:11.930581 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:11.930869 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:11.931200 (+ 331us) server_negotiation.cc:299] Negotiation successful
0925 15:13:11.931458 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":357,"threads_started":1}
W20260925 15:13:11.933032 15714 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:54734
W20260925 15:13:11.934132 12320 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 15591: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 15:13:11.934340 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 15591
I20260925 15:13:11.949371 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 15724
Sep 25 15:13:11 dist-test-slave-p468 krb5kdc[15351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349191, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.12.8.3@KRBTEST.COM
I20260925 15:13:11.964043 15865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:11.950333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46224 (local address 127.12.8.3:34227)
0925 15:13:11.950898 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:11.950904 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:11.950930 (+ 26us) server_negotiation.cc:407] Connection header received
0925 15:13:11.950983 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:11.950988 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:11.951178 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:11.951293 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:11.952480 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:11.953460 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:11.954677 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:11.955202 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:11.955294 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:11.959300 (+ 4006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:11.959310 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:11.959323 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:11.959355 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:11.961797 (+ 2442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:11.962461 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:11.962467 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:11.962470 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:11.962595 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:11.962878 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:11.962881 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:11.962883 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:11.963183 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:11.963388 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:11.963601 (+ 213us) server_negotiation.cc:299] Negotiation successful
0925 15:13:11.963779 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":292,"threads_started":1}
W20260925 15:13:11.965231 15847 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:46224
W20260925 15:13:11.966331 12320 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 15724: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 15:13:11.966495 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 15724
I20260925 15:13:11.981384 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 15364
I20260925 15:13:11.982553 15426 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:13:12.084659 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 15364
2026-09-25T15:13:12Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3529 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349162370577-12320-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:13:12 dist-test-slave-p468 krb5kdc[15869](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:13:12 dist-test-slave-p468 krb5kdc[15869](info): set up 2 sockets
Sep 25 15:13:12 dist-test-slave-p468 krb5kdc[15869](info): commencing operation
krb5kdc: starting...
W20260925 15:13:13.186560 12320 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349162370577-12320-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349162370577-12320-0/krb5kdc/test-user.keytab.
Sep 25 15:13:13 dist-test-slave-p468 krb5kdc[15869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349193, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:13:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:13:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.8.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab.
Entry for principal kudu/127.12.8.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab.
I20260925 15:13:13.374142 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349162370577-12320-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349162370577-12320-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349162370577-12320-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.62:45323
--webserver_interface=127.12.8.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab
--principal=kudu/127.12.8.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:32955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.62:45323
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:13:13.686766 15882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:13:13.687038 15882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:13:13.687193 15882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:13:13.694398 15882 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:13:13.694526 15882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:13:13.694610 15882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:13:13.694711 15882 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:13:13.694787 15882 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:13:13.704332 15882 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:32955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349162370577-12320-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349162370577-12320-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.8.62:45323
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab
--principal=kudu/127.12.8.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.62:45323
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.8.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349162370577-12320-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:13:13.705549 15882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:13:13.706836 15882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:13:13.718272 15888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:13.718315 15890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:13.718354 15887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:13:13.719018 15882 server_base.cc:1034] running on GCE node
I20260925 15:13:13.719730 15882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:13:13.721001 15882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:13:13.722280 15882 hybrid_clock.cc:648] HybridClock initialized: now 1790349193722228 us; error 54 us; skew 500 ppm
I20260925 15:13:13.722620 15882 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:13:13 dist-test-slave-p468 krb5kdc[15869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349193, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:13:13.725739 15882 init.cc:377] Logged in from keytab as kudu/127.12.8.62@KRBTEST.COM (short username kudu)
I20260925 15:13:13.727123 15882 webserver.cc:460] Webserver started at http://127.12.8.62:32943/ using document root <none> and password file <none>
I20260925 15:13:13.727725 15882 fs_manager.cc:359] Metadata directory not provided
I20260925 15:13:13.727821 15882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:13:13.728075 15882 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:13:13.730908 15882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349162370577-12320-0/minicluster-data/master-0/data/instance:
uuid: "bc30157133b441cd96f220c983a04626"
format_stamp: "Formatted at 2026-09-25 15:13:13 on dist-test-slave-p468"
server_key: "e12fad91ab9aec9d07994d66b08c3403"
server_key_iv: "2773e713c8be48faa5bb50f07fc0c71c"
server_key_version: "encryptionkey@0"
I20260925 15:13:13.731633 15882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349162370577-12320-0/minicluster-data/master-0/wal/instance:
uuid: "bc30157133b441cd96f220c983a04626"
format_stamp: "Formatted at 2026-09-25 15:13:13 on dist-test-slave-p468"
server_key: "e12fad91ab9aec9d07994d66b08c3403"
server_key_iv: "2773e713c8be48faa5bb50f07fc0c71c"
server_key_version: "encryptionkey@0"
I20260925 15:13:13.736429 15882 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260925 15:13:13.739562 15897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:13.740825 15882 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 15:13:13.740957 15882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349162370577-12320-0/minicluster-data/master-0/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349162370577-12320-0/minicluster-data/master-0/wal
uuid: "bc30157133b441cd96f220c983a04626"
format_stamp: "Formatted at 2026-09-25 15:13:13 on dist-test-slave-p468"
server_key: "e12fad91ab9aec9d07994d66b08c3403"
server_key_iv: "2773e713c8be48faa5bb50f07fc0c71c"
server_key_version: "encryptionkey@0"
I20260925 15:13:13.741115 15882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349162370577-12320-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349162370577-12320-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349162370577-12320-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:13:13.757185 15882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:13:13.761345 15882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:13:13.761571 15882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:13:13.779515 15882 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.62:45323
I20260925 15:13:13.779548 15948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.62:45323 every 8 connection(s)
I20260925 15:13:13.780901 15882 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
I20260925 15:13:13.784519 15949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:13:13.790997 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 15882
I20260925 15:13:13.791229 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349162370577-12320-0/minicluster-data/master-0/wal/instance
I20260925 15:13:13.791923 12320 external_mini_cluster.cc:1428] Setting key cb0587bb81b0c6b72db3674c9aa61e29
I20260925 15:13:13.794806 15949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc30157133b441cd96f220c983a04626: Bootstrap starting.
I20260925 15:13:13.798138 15949 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc30157133b441cd96f220c983a04626: Neither blocks nor log segments found. Creating new log.
I20260925 15:13:13.799391 15949 log.cc:824] T 00000000000000000000000000000000 P bc30157133b441cd96f220c983a04626: Log is configured to *not* fsync() on all Append() calls
I20260925 15:13:13.802445 15949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc30157133b441cd96f220c983a04626: No bootstrap required, opened a new log
Sep 25 15:13:13 dist-test-slave-p468 krb5kdc[15869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349193, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
I20260925 15:13:13.812783 15949 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc30157133b441cd96f220c983a04626 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc30157133b441cd96f220c983a04626" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 45323 } }
I20260925 15:13:13.813247 15949 raft_consensus.cc:374] T 00000000000000000000000000000000 P bc30157133b441cd96f220c983a04626 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:13:13.813349 15949 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc30157133b441cd96f220c983a04626 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc30157133b441cd96f220c983a04626, State: Initialized, Role: FOLLOWER
I20260925 15:13:13.814016 15949 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc30157133b441cd96f220c983a04626 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc30157133b441cd96f220c983a04626" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 45323 } }
I20260925 15:13:13.814340 15949 raft_consensus.cc:388] T 00000000000000000000000000000000 P bc30157133b441cd96f220c983a04626 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:13:13.814515 15949 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc30157133b441cd96f220c983a04626 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:13:13.814689 15949 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc30157133b441cd96f220c983a04626 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:13:13.816082 15952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:13.793853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60850 (local address 127.12.8.62:45323)
0925 15:13:13.794617 (+ 764us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:13.794629 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:13.794668 (+ 39us) server_negotiation.cc:407] Connection header received
0925 15:13:13.796507 (+ 1839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:13.796528 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:13.796891 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:13.797219 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:13.799025 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:13.800728 (+ 1703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:13.802404 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:13.803138 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:13.803287 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:13.807621 (+ 4334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:13.807631 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:13.807654 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:13.807698 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:13.811557 (+ 3859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:13.812191 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:13.812199 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:13.812207 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:13.812406 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:13.812747 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:13.812752 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:13.812756 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:13.813484 (+ 728us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:13.813768 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:13.814256 (+ 488us) server_negotiation.cc:299] Negotiation successful
0925 15:13:13.814706 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":366,"threads_started":1}
I20260925 15:13:13.817822 15949 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc30157133b441cd96f220c983a04626 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc30157133b441cd96f220c983a04626" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 45323 } }
I20260925 15:13:13.818356 15949 leader_election.cc:302] T 00000000000000000000000000000000 P bc30157133b441cd96f220c983a04626 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc30157133b441cd96f220c983a04626; no voters:
I20260925 15:13:13.818959 15949 leader_election.cc:288] T 00000000000000000000000000000000 P bc30157133b441cd96f220c983a04626 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:13:13.819077 15954 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc30157133b441cd96f220c983a04626 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:13:13.819824 15954 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc30157133b441cd96f220c983a04626 [term 1 LEADER]: Becoming Leader. State: Replica: bc30157133b441cd96f220c983a04626, State: Running, Role: LEADER
I20260925 15:13:13.820356 15954 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc30157133b441cd96f220c983a04626 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc30157133b441cd96f220c983a04626" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 45323 } }
I20260925 15:13:13.820960 15949 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc30157133b441cd96f220c983a04626 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:13:13.824133 15956 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc30157133b441cd96f220c983a04626 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc30157133b441cd96f220c983a04626. Latest consensus state: current_term: 1 leader_uuid: "bc30157133b441cd96f220c983a04626" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc30157133b441cd96f220c983a04626" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 45323 } } }
I20260925 15:13:13.824133 15955 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc30157133b441cd96f220c983a04626 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bc30157133b441cd96f220c983a04626" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc30157133b441cd96f220c983a04626" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 45323 } } }
I20260925 15:13:13.824555 15956 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc30157133b441cd96f220c983a04626 [sys.catalog]: This master's current role is: LEADER
I20260925 15:13:13.824560 15955 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc30157133b441cd96f220c983a04626 [sys.catalog]: This master's current role is: LEADER
I20260925 15:13:13.825388 15962 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:13:13.830447 15962 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:13:13.838960 15962 catalog_manager.cc:1347] Generated new cluster ID: e7a3652d33264c109e1dc916a312af07
I20260925 15:13:13.839084 15962 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:13:13.848553 15962 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:13:13.849746 15962 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:13:13.867975 15962 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bc30157133b441cd96f220c983a04626: Generated new TSK 0
I20260925 15:13:13.868748 15962 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.8.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.1.keytab.
Entry for principal kudu/127.12.8.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.1.keytab.
I20260925 15:13:13.921518 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349162370577-12320-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349162370577-12320-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349162370577-12320-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349162370577-12320-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.1:0
--local_ip_for_outbound_sockets=127.12.8.1
--webserver_interface=127.12.8.1
--webserver_port=0
--tserver_master_addrs=127.12.8.62:45323
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.1.keytab
--principal=kudu/127.12.8.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:32955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:13:14.253583 15975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:13:14.253891 15975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:13:14.254056 15975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:13:14.261507 15975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:13:14.261646 15975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:13:14.261842 15975 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.1
I20260925 15:13:14.271468 15975 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:32955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349162370577-12320-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349162370577-12320-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.1.keytab
--principal=kudu/127.12.8.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349162370577-12320-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.8.1
--webserver_port=0
--tserver_master_addrs=127.12.8.62:45323
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.1
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349162370577-12320-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:13:14.272682 15975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:13:14.273986 15975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:13:14.286815 15980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:14.286815 15981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:14.287611 15983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:13:14.287648 15975 server_base.cc:1034] running on GCE node
I20260925 15:13:14.288245 15975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:13:14.289140 15975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:13:14.290355 15975 hybrid_clock.cc:648] HybridClock initialized: now 1790349194290313 us; error 51 us; skew 500 ppm
I20260925 15:13:14.290690 15975 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:13:14 dist-test-slave-p468 krb5kdc[15869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349194, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:13:14.293852 15975 init.cc:377] Logged in from keytab as kudu/127.12.8.1@KRBTEST.COM (short username kudu)
I20260925 15:13:14.295301 15975 webserver.cc:460] Webserver started at http://127.12.8.1:40203/ using document root <none> and password file <none>
I20260925 15:13:14.295955 15975 fs_manager.cc:359] Metadata directory not provided
I20260925 15:13:14.296099 15975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:13:14.296437 15975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:13:14.299243 15975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349162370577-12320-0/minicluster-data/ts-0/data/instance:
uuid: "27a114bbd4044dd29740cd3877e4f5ad"
format_stamp: "Formatted at 2026-09-25 15:13:14 on dist-test-slave-p468"
server_key: "ce54b276a735b5fc351612bdbf7202ad"
server_key_iv: "6506c44b1bb52d4347630725c32d7788"
server_key_version: "encryptionkey@0"
I20260925 15:13:14.300037 15975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349162370577-12320-0/minicluster-data/ts-0/wal/instance:
uuid: "27a114bbd4044dd29740cd3877e4f5ad"
format_stamp: "Formatted at 2026-09-25 15:13:14 on dist-test-slave-p468"
server_key: "ce54b276a735b5fc351612bdbf7202ad"
server_key_iv: "6506c44b1bb52d4347630725c32d7788"
server_key_version: "encryptionkey@0"
I20260925 15:13:14.304834 15975 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260925 15:13:14.308076 15990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:14.309312 15975 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 15:13:14.309477 15975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349162370577-12320-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349162370577-12320-0/minicluster-data/ts-0/wal
uuid: "27a114bbd4044dd29740cd3877e4f5ad"
format_stamp: "Formatted at 2026-09-25 15:13:14 on dist-test-slave-p468"
server_key: "ce54b276a735b5fc351612bdbf7202ad"
server_key_iv: "6506c44b1bb52d4347630725c32d7788"
server_key_version: "encryptionkey@0"
I20260925 15:13:14.309681 15975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349162370577-12320-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349162370577-12320-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349162370577-12320-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:13:14.335870 15975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:13:14.340186 15975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:13:14.340471 15975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:13:14.341399 15975 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:13:14.342710 15975 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:13:14.342808 15975 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:14.342950 15975 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:13:14.343021 15975 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:14.375583 15975 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.1:37141
I20260925 15:13:14.375630 16102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.1:37141 every 8 connection(s)
I20260925 15:13:14.376819 15975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349162370577-12320-0/minicluster-data/ts-0/data/info.pb
I20260925 15:13:14.385190 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 15975
I20260925 15:13:14.385494 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349162370577-12320-0/minicluster-data/ts-0/wal/instance
I20260925 15:13:14.386125 12320 external_mini_cluster.cc:1428] Setting key e47e985c8d1f9fd61f3c389795582887
Sep 25 15:13:14 dist-test-slave-p468 krb5kdc[15869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349194, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.1@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
I20260925 15:13:14.396937 16105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:14.379443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.1:50817 (local address 127.12.8.62:45323)
0925 15:13:14.379892 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:14.379897 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:14.380732 (+ 835us) server_negotiation.cc:407] Connection header received
0925 15:13:14.382594 (+ 1862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:14.382600 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:14.382685 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:14.382799 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:14.384243 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:14.384968 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:14.386694 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:14.387119 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:14.387286 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:14.391285 (+ 3999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:14.391290 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:14.391293 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:14.391325 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:14.393894 (+ 2569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:14.394706 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:14.394710 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:14.394713 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:14.394824 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:14.395248 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:14.395252 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:14.395256 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:14.395525 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:14.395676 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:14.396366 (+ 690us) server_negotiation.cc:299] Negotiation successful
0925 15:13:14.396588 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":254,"threads_started":1}
I20260925 15:13:14.397857 16106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:14.379906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:45323 (local address 127.12.8.1:50817)
0925 15:13:14.380578 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:14.380619 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:14.382326 (+ 1707us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:14.383027 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:14.383035 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:14.383530 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:13:14.384076 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:14.384092 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:14.385128 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:14.385133 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:14.386553 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:14.386569 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:14.387400 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:14.387406 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:14.387612 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:14.388392 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:13:14.388421 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:13:14.391058 (+ 2637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:13:14.394045 (+ 2987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:13:14.394054 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:13:14.394070 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:13:14.394573 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:13:14.394928 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:13:14.394934 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:13:14.394937 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:13:14.395128 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:13:14.395649 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:13:14.395657 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:13:14.395990 (+ 333us) client_negotiation.cc:769] Sending connection context
0925 15:13:14.396251 (+ 261us) client_negotiation.cc:240] Negotiation successful
0925 15:13:14.396753 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":262,"threads_started":1}
I20260925 15:13:14.399385 16103 heartbeater.cc:347] Connected to a master server at 127.12.8.62:45323
I20260925 15:13:14.399704 16103 heartbeater.cc:464] Registering TS with master...
I20260925 15:13:14.400585 16103 heartbeater.cc:511] Master 127.12.8.62:45323 requested a full tablet report, sending...
I20260925 15:13:14.402788 15914 ts_manager.cc:194] Registered new tserver with Master: 27a114bbd4044dd29740cd3877e4f5ad (127.12.8.1:37141)
I20260925 15:13:14.404187 15914 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.1@KRBTEST.COM'} at 127.12.8.1:50817
WARNING: no policy specified for kudu/127.12.8.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.2.keytab.
Entry for principal kudu/127.12.8.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.2.keytab.
I20260925 15:13:14.429684 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349162370577-12320-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349162370577-12320-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349162370577-12320-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349162370577-12320-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.2:0
--local_ip_for_outbound_sockets=127.12.8.2
--webserver_interface=127.12.8.2
--webserver_port=0
--tserver_master_addrs=127.12.8.62:45323
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.2.keytab
--principal=kudu/127.12.8.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:32955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:13:14.734119 16109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:13:14.734473 16109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:13:14.734632 16109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:13:14.742255 16109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:13:14.742385 16109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:13:14.742565 16109 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.2
I20260925 15:13:14.751989 16109 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:32955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349162370577-12320-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349162370577-12320-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.2.keytab
--principal=kudu/127.12.8.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349162370577-12320-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.8.2
--webserver_port=0
--tserver_master_addrs=127.12.8.62:45323
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.2
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349162370577-12320-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:13:14.753183 16109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:13:14.754482 16109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:13:14.767267 16115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:14.767665 16114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:14.767716 16117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:13:14.768342 16109 server_base.cc:1034] running on GCE node
I20260925 15:13:14.768918 16109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:13:14.769833 16109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:13:14.771051 16109 hybrid_clock.cc:648] HybridClock initialized: now 1790349194770992 us; error 56 us; skew 500 ppm
I20260925 15:13:14.771358 16109 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:13:14 dist-test-slave-p468 krb5kdc[15869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349194, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:13:14.774396 16109 init.cc:377] Logged in from keytab as kudu/127.12.8.2@KRBTEST.COM (short username kudu)
I20260925 15:13:14.775775 16109 webserver.cc:460] Webserver started at http://127.12.8.2:34337/ using document root <none> and password file <none>
I20260925 15:13:14.776381 16109 fs_manager.cc:359] Metadata directory not provided
I20260925 15:13:14.776475 16109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:13:14.776727 16109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:13:14.779610 16109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349162370577-12320-0/minicluster-data/ts-1/data/instance:
uuid: "ed0fa6a7c4a3468d9a7363f8a00ff26d"
format_stamp: "Formatted at 2026-09-25 15:13:14 on dist-test-slave-p468"
server_key: "205d5ab4bb91d31d8d62741cb93e21b4"
server_key_iv: "507825c118728ac4debd3dae2f03b661"
server_key_version: "encryptionkey@0"
I20260925 15:13:14.780364 16109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349162370577-12320-0/minicluster-data/ts-1/wal/instance:
uuid: "ed0fa6a7c4a3468d9a7363f8a00ff26d"
format_stamp: "Formatted at 2026-09-25 15:13:14 on dist-test-slave-p468"
server_key: "205d5ab4bb91d31d8d62741cb93e21b4"
server_key_iv: "507825c118728ac4debd3dae2f03b661"
server_key_version: "encryptionkey@0"
I20260925 15:13:14.785382 16109 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260925 15:13:14.788770 16124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:14.790042 16109 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 15:13:14.790283 16109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349162370577-12320-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349162370577-12320-0/minicluster-data/ts-1/wal
uuid: "ed0fa6a7c4a3468d9a7363f8a00ff26d"
format_stamp: "Formatted at 2026-09-25 15:13:14 on dist-test-slave-p468"
server_key: "205d5ab4bb91d31d8d62741cb93e21b4"
server_key_iv: "507825c118728ac4debd3dae2f03b661"
server_key_version: "encryptionkey@0"
I20260925 15:13:14.790480 16109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349162370577-12320-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349162370577-12320-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349162370577-12320-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:13:14.815654 16109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:13:14.819751 16109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:13:14.820043 16109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:13:14.821038 16109 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:13:14.822538 16109 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:13:14.822639 16109 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:14.822781 16109 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:13:14.822852 16109 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:14.854702 16109 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.2:44439
I20260925 15:13:14.854751 16236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.2:44439 every 8 connection(s)
I20260925 15:13:14.855942 16109 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349162370577-12320-0/minicluster-data/ts-1/data/info.pb
I20260925 15:13:14.856360 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 16109
I20260925 15:13:14.856562 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349162370577-12320-0/minicluster-data/ts-1/wal/instance
I20260925 15:13:14.857117 12320 external_mini_cluster.cc:1428] Setting key 0a77709e91bbf937a7485e3693140b9e
Sep 25 15:13:14 dist-test-slave-p468 krb5kdc[15869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349194, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.2@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.8.3@KRBTEST.COM; defaulting to no policy
I20260925 15:13:14.878746 16105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:14.858857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.2:57185 (local address 127.12.8.62:45323)
0925 15:13:14.859028 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:14.859033 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:14.860384 (+ 1351us) server_negotiation.cc:407] Connection header received
0925 15:13:14.863171 (+ 2787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:14.863175 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:14.863237 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:14.863320 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:14.865393 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:14.866091 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:14.868674 (+ 2583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:14.869083 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:14.869182 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:14.873178 (+ 3996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:14.873183 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:14.873186 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:14.873215 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:14.875753 (+ 2538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:14.876450 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:14.876457 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:14.876465 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:14.876581 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:14.876995 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:14.877000 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:14.877003 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:14.877299 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:14.877449 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:14.878175 (+ 726us) server_negotiation.cc:299] Negotiation successful
0925 15:13:14.878410 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.3@KRBTEST.COM" created.
I20260925 15:13:14.879796 16239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:14.859384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:45323 (local address 127.12.8.2:57185)
0925 15:13:14.860209 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:14.860267 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:14.862897 (+ 2630us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:14.863577 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:14.863590 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:14.864350 (+ 760us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:13:14.865138 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:14.865167 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:14.866261 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:14.866270 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:14.868523 (+ 2253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:14.868541 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:14.869304 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:14.869313 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:14.869635 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:14.870395 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:13:14.870428 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:13:14.872968 (+ 2540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:13:14.875890 (+ 2922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:13:14.875899 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:13:14.875914 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:13:14.876324 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:13:14.876693 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:13:14.876698 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:13:14.876701 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:13:14.876876 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:13:14.877442 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:13:14.877449 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:13:14.877759 (+ 310us) client_negotiation.cc:769] Sending connection context
0925 15:13:14.878072 (+ 313us) client_negotiation.cc:240] Negotiation successful
0925 15:13:14.878605 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":313,"threads_started":1}
I20260925 15:13:14.881234 16237 heartbeater.cc:347] Connected to a master server at 127.12.8.62:45323
I20260925 15:13:14.881593 16237 heartbeater.cc:464] Registering TS with master...
I20260925 15:13:14.882369 16237 heartbeater.cc:511] Master 127.12.8.62:45323 requested a full tablet report, sending...
I20260925 15:13:14.883839 15914 ts_manager.cc:194] Registered new tserver with Master: ed0fa6a7c4a3468d9a7363f8a00ff26d (127.12.8.2:44439)
I20260925 15:13:14.884634 15914 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.2@KRBTEST.COM'} at 127.12.8.2:57185
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.3.keytab.
Entry for principal kudu/127.12.8.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.3.keytab.
I20260925 15:13:14.901322 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349162370577-12320-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349162370577-12320-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349162370577-12320-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349162370577-12320-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.3:0
--local_ip_for_outbound_sockets=127.12.8.3
--webserver_interface=127.12.8.3
--webserver_port=0
--tserver_master_addrs=127.12.8.62:45323
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.3.keytab
--principal=kudu/127.12.8.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:32955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:13:15.210238 16242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:13:15.210517 16242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:13:15.210680 16242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:13:15.217955 16242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:13:15.218084 16242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:13:15.218290 16242 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.3
I20260925 15:13:15.228086 16242 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:32955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349162370577-12320-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349162370577-12320-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.3.keytab
--principal=kudu/127.12.8.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349162370577-12320-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.8.3
--webserver_port=0
--tserver_master_addrs=127.12.8.62:45323
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.3
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349162370577-12320-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:13:15.229328 16242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:13:15.230585 16242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:13:15.243505 16250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:15.243536 16248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:15.243511 16247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:13:15.244733 16242 server_base.cc:1034] running on GCE node
I20260925 15:13:15.245402 16242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:13:15.246452 16242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:13:15.247678 16242 hybrid_clock.cc:648] HybridClock initialized: now 1790349195247603 us; error 65 us; skew 500 ppm
I20260925 15:13:15.247958 16242 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:13:15 dist-test-slave-p468 krb5kdc[15869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349195, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:13:15.256175 16242 init.cc:377] Logged in from keytab as kudu/127.12.8.3@KRBTEST.COM (short username kudu)
I20260925 15:13:15.257606 16242 webserver.cc:460] Webserver started at http://127.12.8.3:36061/ using document root <none> and password file <none>
I20260925 15:13:15.258340 16242 fs_manager.cc:359] Metadata directory not provided
I20260925 15:13:15.258484 16242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:13:15.258795 16242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:13:15.261605 16242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349162370577-12320-0/minicluster-data/ts-2/data/instance:
uuid: "7183f29494294d1081d711c63e0d4441"
format_stamp: "Formatted at 2026-09-25 15:13:15 on dist-test-slave-p468"
server_key: "261a072137cd9f91e750a7ec9a116382"
server_key_iv: "fa475eed176a2d5619e437d488221b7b"
server_key_version: "encryptionkey@0"
I20260925 15:13:15.262450 16242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349162370577-12320-0/minicluster-data/ts-2/wal/instance:
uuid: "7183f29494294d1081d711c63e0d4441"
format_stamp: "Formatted at 2026-09-25 15:13:15 on dist-test-slave-p468"
server_key: "261a072137cd9f91e750a7ec9a116382"
server_key_iv: "fa475eed176a2d5619e437d488221b7b"
server_key_version: "encryptionkey@0"
I20260925 15:13:15.267479 16242 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260925 15:13:15.270783 16257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:15.271962 16242 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260925 15:13:15.272145 16242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349162370577-12320-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349162370577-12320-0/minicluster-data/ts-2/wal
uuid: "7183f29494294d1081d711c63e0d4441"
format_stamp: "Formatted at 2026-09-25 15:13:15 on dist-test-slave-p468"
server_key: "261a072137cd9f91e750a7ec9a116382"
server_key_iv: "fa475eed176a2d5619e437d488221b7b"
server_key_version: "encryptionkey@0"
I20260925 15:13:15.272337 16242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349162370577-12320-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349162370577-12320-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349162370577-12320-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:13:15.295302 16242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:13:15.299592 16242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:13:15.299876 16242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:13:15.300766 16242 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:13:15.302049 16242 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:13:15.302155 16242 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:15.302326 16242 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:13:15.302398 16242 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:15.334960 16242 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.3:40499
I20260925 15:13:15.335047 16369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.3:40499 every 8 connection(s)
I20260925 15:13:15.336253 16242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349162370577-12320-0/minicluster-data/ts-2/data/info.pb
I20260925 15:13:15.337956 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 16242
I20260925 15:13:15.338157 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349162370577-12320-0/minicluster-data/ts-2/wal/instance
I20260925 15:13:15.338598 12320 external_mini_cluster.cc:1428] Setting key 0c302d0b1de7b5bbcd7a8dc6b03b49a8
Sep 25 15:13:15 dist-test-slave-p468 krb5kdc[15869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349195, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.3@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
I20260925 15:13:15.355531 16105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:15.338780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.3:35045 (local address 127.12.8.62:45323)
0925 15:13:15.338986 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:15.338992 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:15.340357 (+ 1365us) server_negotiation.cc:407] Connection header received
0925 15:13:15.342506 (+ 2149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:15.342510 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:15.342571 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:15.342653 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:15.344009 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:15.344816 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:15.346386 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:15.346759 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:15.346848 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:15.350069 (+ 3221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:15.350074 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:15.350077 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:15.350107 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:15.352770 (+ 2663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:15.353630 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:15.353634 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:15.353636 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:15.353751 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:15.354122 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:15.354125 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:15.354127 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:15.354414 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:15.354520 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:15.355141 (+ 621us) server_negotiation.cc:299] Negotiation successful
0925 15:13:15.355302 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 15:13:15.356905 16372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:15.339338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:45323 (local address 127.12.8.3:35045)
0925 15:13:15.340204 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:15.340258 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:15.342267 (+ 2009us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:15.342843 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:15.342851 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:15.343317 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:13:15.343871 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:15.343887 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:15.344934 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:15.344938 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:15.346269 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:15.346280 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:15.346929 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:15.346933 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:15.347117 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:15.347781 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:13:15.347802 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:13:15.349880 (+ 2078us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:13:15.352893 (+ 3013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:13:15.352899 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:13:15.352911 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:13:15.353491 (+ 580us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:13:15.353843 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:13:15.353846 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:13:15.353848 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:13:15.354033 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:13:15.354517 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:13:15.354522 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:13:15.354809 (+ 287us) client_negotiation.cc:769] Sending connection context
0925 15:13:15.355051 (+ 242us) client_negotiation.cc:240] Negotiation successful
0925 15:13:15.355404 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":306,"threads_started":1}
I20260925 15:13:15.358264 16370 heartbeater.cc:347] Connected to a master server at 127.12.8.62:45323
I20260925 15:13:15.358572 16370 heartbeater.cc:464] Registering TS with master...
I20260925 15:13:15.359210 16370 heartbeater.cc:511] Master 127.12.8.62:45323 requested a full tablet report, sending...
I20260925 15:13:15.360538 15914 ts_manager.cc:194] Registered new tserver with Master: 7183f29494294d1081d711c63e0d4441 (127.12.8.3:40499)
I20260925 15:13:15.361241 15914 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.3@KRBTEST.COM'} at 127.12.8.3:35045
I20260925 15:13:15.365533 12320 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 15:13:15 dist-test-slave-p468 krb5kdc[15869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 15:13:15 dist-test-slave-p468 krb5kdc[15869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.1@KRBTEST.COM
I20260925 15:13:15.406978 16103 heartbeater.cc:503] Master 127.12.8.62:45323 was elected leader, sending a full tablet report...
I20260925 15:13:15.413123 16377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:15.398148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43348 (local address 127.12.8.1:37141)
0925 15:13:15.399014 (+ 866us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:15.399021 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:15.399045 (+ 24us) server_negotiation.cc:407] Connection header received
0925 15:13:15.399102 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:15.399108 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:15.399312 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:15.399484 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:15.400881 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:15.401882 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:15.403184 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:15.403744 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:15.403877 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:15.407775 (+ 3898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:15.407786 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:15.407799 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:15.407827 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:15.410731 (+ 2904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:15.411329 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:15.411335 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:15.411337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:15.411452 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:15.411749 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:15.411752 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:15.411754 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:15.412056 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:15.412256 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:15.412479 (+ 223us) server_negotiation.cc:299] Negotiation successful
0925 15:13:15.412748 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":679,"thread_start_us":493,"threads_started":1}
Sep 25 15:13:15 dist-test-slave-p468 krb5kdc[15869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
I20260925 15:13:15.430112 16105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:15.417143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60864 (local address 127.12.8.62:45323)
0925 15:13:15.417351 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:15.417355 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:15.417460 (+ 105us) server_negotiation.cc:407] Connection header received
0925 15:13:15.417594 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:15.417597 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:15.417656 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:15.417732 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:15.418902 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:15.419595 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:15.420728 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:15.421248 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:15.421338 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:15.425054 (+ 3716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:15.425059 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:15.425062 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:15.425092 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:15.427951 (+ 2859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:15.428501 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:15.428505 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:15.428507 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:15.428609 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:15.428893 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:15.428896 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:15.428898 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:15.429197 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:15.429370 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:15.429627 (+ 257us) server_negotiation.cc:299] Negotiation successful
0925 15:13:15.429776 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260925 15:13:15.430696 15914 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60864
I20260925 15:13:15.431658 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 15975
I20260925 15:13:15.442853 16377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:15.432454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43364 (local address 127.12.8.1:37141)
0925 15:13:15.432695 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:15.432701 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:15.432851 (+ 150us) server_negotiation.cc:407] Connection header received
0925 15:13:15.432964 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:15.432967 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:15.433059 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:15.433137 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:15.434117 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:15.434845 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:15.435988 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:15.436371 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:15.436462 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:15.438581 (+ 2119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:15.438585 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:15.438589 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:15.438616 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:15.440815 (+ 2199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:15.441342 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:15.441345 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:15.441348 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:15.441451 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:15.441743 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:15.441745 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:15.441747 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:15.441950 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:15.442117 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:15.442381 (+ 264us) server_negotiation.cc:299] Negotiation successful
0925 15:13:15.442524 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260925 15:13:15.443404 16098 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:13:15.525210 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 15975
I20260925 15:13:15.542614 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 16109
Sep 25 15:13:15 dist-test-slave-p468 krb5kdc[15869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.2@KRBTEST.COM
I20260925 15:13:15.557890 16382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:15.543501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43230 (local address 127.12.8.2:44439)
0925 15:13:15.543999 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:15.544007 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:15.544030 (+ 23us) server_negotiation.cc:407] Connection header received
0925 15:13:15.544088 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:15.544094 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:15.544294 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:15.544480 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:15.545651 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:15.546730 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:15.547956 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:15.548496 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:15.548639 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:15.552815 (+ 4176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:15.552826 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:15.552840 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:15.552873 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:15.555550 (+ 2677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:15.556141 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:15.556147 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:15.556149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:15.556262 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:15.556563 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:15.556566 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:15.556568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:15.556884 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:15.557101 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:15.557315 (+ 214us) server_negotiation.cc:299] Negotiation successful
0925 15:13:15.557540 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":261,"threads_started":1}
I20260925 15:13:15.559162 16232 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:13:15.640065 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 16109
I20260925 15:13:15.655282 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 16242
Sep 25 15:13:15 dist-test-slave-p468 krb5kdc[15869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.3@KRBTEST.COM
I20260925 15:13:15.671485 16384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:15.656499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52190 (local address 127.12.8.3:40499)
0925 15:13:15.657124 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:15.657130 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:15.657159 (+ 29us) server_negotiation.cc:407] Connection header received
0925 15:13:15.657214 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:15.657220 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:15.657421 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:15.657557 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:15.658850 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:15.659843 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:15.661102 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:15.661632 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:15.661724 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:15.666356 (+ 4632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:15.666377 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:15.666390 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:15.666428 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:15.669032 (+ 2604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:15.669621 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:15.669627 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:15.669629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:15.669742 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:15.670056 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:15.670059 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:15.670061 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:15.670434 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:15.670661 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:15.670911 (+ 250us) server_negotiation.cc:299] Negotiation successful
0925 15:13:15.671097 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":336,"threads_started":1}
I20260925 15:13:15.672806 16365 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:13:15.756992 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 16242
I20260925 15:13:15.772329 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 15882
I20260925 15:13:15.773501 15944 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:13:15.877302 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 15882
2026-09-25T15:13:15Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3793 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349162370577-12320-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:13:15 dist-test-slave-p468 krb5kdc[16389](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:13:15 dist-test-slave-p468 krb5kdc[16389](info): set up 2 sockets
Sep 25 15:13:15 dist-test-slave-p468 krb5kdc[16389](info): commencing operation
krb5kdc: starting...
W20260925 15:13:16.954665 12320 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.002s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349162370577-12320-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349162370577-12320-0/krb5kdc/test-user.keytab.
Sep 25 15:13:17 dist-test-slave-p468 krb5kdc[16389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349197, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:13:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:13:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.8.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab.
Entry for principal kudu/127.12.8.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab.
I20260925 15:13:17.138106 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349162370577-12320-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349162370577-12320-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349162370577-12320-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.62:41241
--webserver_interface=127.12.8.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab
--principal=kudu/127.12.8.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:46795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.62:41241
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:13:17.446969 16402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:13:17.447299 16402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:13:17.447458 16402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:13:17.454646 16402 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:13:17.454783 16402 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:13:17.454885 16402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:13:17.454985 16402 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:13:17.455065 16402 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:13:17.464718 16402 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:46795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349162370577-12320-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349162370577-12320-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.8.62:41241
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab
--principal=kudu/127.12.8.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.62:41241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.8.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349162370577-12320-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:13:17.465972 16402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:13:17.467291 16402 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:13:17.478475 16407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:17.478540 16410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:17.478564 16408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:13:17.479550 16402 server_base.cc:1034] running on GCE node
I20260925 15:13:17.480351 16402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:13:17.481700 16402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:13:17.482900 16402 hybrid_clock.cc:648] HybridClock initialized: now 1790349197482859 us; error 43 us; skew 500 ppm
I20260925 15:13:17.483249 16402 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:13:17 dist-test-slave-p468 krb5kdc[16389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349197, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:13:17.486178 16402 init.cc:377] Logged in from keytab as kudu/127.12.8.62@KRBTEST.COM (short username kudu)
I20260925 15:13:17.487340 16402 fs_manager.cc:359] Metadata directory not provided
I20260925 15:13:17.487475 16402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:13:17.487789 16402 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:13:17.490702 16402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349162370577-12320-0/minicluster-data/master-0/data/instance:
uuid: "9dbb7b4f6abe47d091c220670710851c"
format_stamp: "Formatted at 2026-09-25 15:13:17 on dist-test-slave-p468"
server_key: "fb1cb9fd099d5c1943ec015d6bb59c8e"
server_key_iv: "1fa3c4c7c3d9811168a481e4071010cb"
server_key_version: "encryptionkey@0"
I20260925 15:13:17.491503 16402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349162370577-12320-0/minicluster-data/master-0/wal/instance:
uuid: "9dbb7b4f6abe47d091c220670710851c"
format_stamp: "Formatted at 2026-09-25 15:13:17 on dist-test-slave-p468"
server_key: "fb1cb9fd099d5c1943ec015d6bb59c8e"
server_key_iv: "1fa3c4c7c3d9811168a481e4071010cb"
server_key_version: "encryptionkey@0"
I20260925 15:13:17.496347 16402 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 15:13:17.499629 16416 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:17.500838 16402 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 15:13:17.501021 16402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349162370577-12320-0/minicluster-data/master-0/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349162370577-12320-0/minicluster-data/master-0/wal
uuid: "9dbb7b4f6abe47d091c220670710851c"
format_stamp: "Formatted at 2026-09-25 15:13:17 on dist-test-slave-p468"
server_key: "fb1cb9fd099d5c1943ec015d6bb59c8e"
server_key_iv: "1fa3c4c7c3d9811168a481e4071010cb"
server_key_version: "encryptionkey@0"
I20260925 15:13:17.501225 16402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349162370577-12320-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349162370577-12320-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349162370577-12320-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:13:17.531810 16402 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:13:17.535676 16402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:13:17.535949 16402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:13:17.554136 16402 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.62:41241
I20260925 15:13:17.554157 16467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.62:41241 every 8 connection(s)
I20260925 15:13:17.555073 16402 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
I20260925 15:13:17.558775 16468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:13:17.564011 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 16402
I20260925 15:13:17.564229 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349162370577-12320-0/minicluster-data/master-0/wal/instance
I20260925 15:13:17.564983 12320 external_mini_cluster.cc:1428] Setting key d13693d723b7763369c62b77419fb6a4
I20260925 15:13:17.569789 16468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dbb7b4f6abe47d091c220670710851c: Bootstrap starting.
I20260925 15:13:17.572690 16468 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9dbb7b4f6abe47d091c220670710851c: Neither blocks nor log segments found. Creating new log.
I20260925 15:13:17.573694 16468 log.cc:824] T 00000000000000000000000000000000 P 9dbb7b4f6abe47d091c220670710851c: Log is configured to *not* fsync() on all Append() calls
I20260925 15:13:17.576617 16468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dbb7b4f6abe47d091c220670710851c: No bootstrap required, opened a new log
Sep 25 15:13:17 dist-test-slave-p468 krb5kdc[16389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349197, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
I20260925 15:13:17.586539 16468 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9dbb7b4f6abe47d091c220670710851c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dbb7b4f6abe47d091c220670710851c" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 41241 } }
I20260925 15:13:17.586952 16468 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9dbb7b4f6abe47d091c220670710851c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:13:17.587054 16468 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9dbb7b4f6abe47d091c220670710851c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dbb7b4f6abe47d091c220670710851c, State: Initialized, Role: FOLLOWER
I20260925 15:13:17.587699 16468 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9dbb7b4f6abe47d091c220670710851c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dbb7b4f6abe47d091c220670710851c" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 41241 } }
I20260925 15:13:17.587962 16468 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9dbb7b4f6abe47d091c220670710851c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:13:17.588125 16471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:17.567021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35344 (local address 127.12.8.62:41241)
0925 15:13:17.567763 (+ 742us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:17.567774 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:17.567815 (+ 41us) server_negotiation.cc:407] Connection header received
0925 15:13:17.569412 (+ 1597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:17.569433 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:17.569789 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:17.570234 (+ 445us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:17.571669 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:17.572996 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:17.574405 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:17.575264 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:17.575446 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:17.579580 (+ 4134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:17.579590 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:17.579607 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:17.579656 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:17.583707 (+ 4051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:17.584481 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:17.584495 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:17.584505 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:17.584712 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:17.585075 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:17.585081 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:17.585084 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:17.585772 (+ 688us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:17.586047 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:17.586404 (+ 357us) server_negotiation.cc:299] Negotiation successful
0925 15:13:17.586808 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":345,"threads_started":1}
I20260925 15:13:17.588122 16468 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9dbb7b4f6abe47d091c220670710851c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:13:17.588333 16468 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9dbb7b4f6abe47d091c220670710851c [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:13:17.591210 16468 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9dbb7b4f6abe47d091c220670710851c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dbb7b4f6abe47d091c220670710851c" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 41241 } }
I20260925 15:13:17.591668 16468 leader_election.cc:302] T 00000000000000000000000000000000 P 9dbb7b4f6abe47d091c220670710851c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9dbb7b4f6abe47d091c220670710851c; no voters:
I20260925 15:13:17.592137 16468 leader_election.cc:288] T 00000000000000000000000000000000 P 9dbb7b4f6abe47d091c220670710851c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:13:17.592295 16473 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9dbb7b4f6abe47d091c220670710851c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:13:17.593400 16473 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9dbb7b4f6abe47d091c220670710851c [term 1 LEADER]: Becoming Leader. State: Replica: 9dbb7b4f6abe47d091c220670710851c, State: Running, Role: LEADER
I20260925 15:13:17.594043 16473 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9dbb7b4f6abe47d091c220670710851c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dbb7b4f6abe47d091c220670710851c" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 41241 } }
I20260925 15:13:17.594251 16468 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9dbb7b4f6abe47d091c220670710851c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:13:17.596551 16475 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dbb7b4f6abe47d091c220670710851c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9dbb7b4f6abe47d091c220670710851c. Latest consensus state: current_term: 1 leader_uuid: "9dbb7b4f6abe47d091c220670710851c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dbb7b4f6abe47d091c220670710851c" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 41241 } } }
I20260925 15:13:17.596906 16475 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dbb7b4f6abe47d091c220670710851c [sys.catalog]: This master's current role is: LEADER
I20260925 15:13:17.597436 16474 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dbb7b4f6abe47d091c220670710851c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9dbb7b4f6abe47d091c220670710851c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dbb7b4f6abe47d091c220670710851c" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 41241 } } }
I20260925 15:13:17.597811 16474 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dbb7b4f6abe47d091c220670710851c [sys.catalog]: This master's current role is: LEADER
I20260925 15:13:17.598016 16480 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:13:17.609723 16480 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:13:17.619735 16480 catalog_manager.cc:1347] Generated new cluster ID: 879e9b2793834d2d9c10230c16bd92d8
I20260925 15:13:17.619837 16480 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:13:17.643895 16480 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:13:17.645370 16480 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:13:17.655642 16480 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9dbb7b4f6abe47d091c220670710851c: Generated new TSK 0
I20260925 15:13:17.656576 16480 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.8.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.1.keytab.
Entry for principal kudu/127.12.8.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.1.keytab.
I20260925 15:13:17.715796 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349162370577-12320-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349162370577-12320-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349162370577-12320-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349162370577-12320-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.1:0
--local_ip_for_outbound_sockets=127.12.8.1
--webserver_interface=127.12.8.1
--webserver_port=0
--tserver_master_addrs=127.12.8.62:41241
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.1.keytab
--principal=kudu/127.12.8.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:46795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:13:18.039024 16495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:13:18.039379 16495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:13:18.039549 16495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:13:18.046566 16495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:13:18.046705 16495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:13:18.046890 16495 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.1
I20260925 15:13:18.056268 16495 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:46795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349162370577-12320-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349162370577-12320-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.1.keytab
--principal=kudu/127.12.8.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349162370577-12320-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.8.1
--webserver_port=0
--tserver_master_addrs=127.12.8.62:41241
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.1
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349162370577-12320-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:13:18.057547 16495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:13:18.058872 16495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:13:18.071203 16500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:18.071238 16503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:18.071306 16501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:13:18.071906 16495 server_base.cc:1034] running on GCE node
I20260925 15:13:18.072461 16495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:13:18.073324 16495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:13:18.074579 16495 hybrid_clock.cc:648] HybridClock initialized: now 1790349198074529 us; error 37 us; skew 500 ppm
I20260925 15:13:18.074930 16495 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:13:18 dist-test-slave-p468 krb5kdc[16389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349198, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:13:18.078006 16495 init.cc:377] Logged in from keytab as kudu/127.12.8.1@KRBTEST.COM (short username kudu)
I20260925 15:13:18.079208 16495 fs_manager.cc:359] Metadata directory not provided
I20260925 15:13:18.079339 16495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:13:18.079679 16495 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:13:18.082475 16495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349162370577-12320-0/minicluster-data/ts-0/data/instance:
uuid: "6bc48e92ec99412f877fd1a4d5770edd"
format_stamp: "Formatted at 2026-09-25 15:13:18 on dist-test-slave-p468"
server_key: "028e6a125a69872e6045149ec71eddd4"
server_key_iv: "78b92b68f49ca0f596a32348602eff73"
server_key_version: "encryptionkey@0"
I20260925 15:13:18.083266 16495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349162370577-12320-0/minicluster-data/ts-0/wal/instance:
uuid: "6bc48e92ec99412f877fd1a4d5770edd"
format_stamp: "Formatted at 2026-09-25 15:13:18 on dist-test-slave-p468"
server_key: "028e6a125a69872e6045149ec71eddd4"
server_key_iv: "78b92b68f49ca0f596a32348602eff73"
server_key_version: "encryptionkey@0"
I20260925 15:13:18.088176 16495 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 15:13:18.091562 16509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:18.092983 16495 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 15:13:18.093169 16495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349162370577-12320-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349162370577-12320-0/minicluster-data/ts-0/wal
uuid: "6bc48e92ec99412f877fd1a4d5770edd"
format_stamp: "Formatted at 2026-09-25 15:13:18 on dist-test-slave-p468"
server_key: "028e6a125a69872e6045149ec71eddd4"
server_key_iv: "78b92b68f49ca0f596a32348602eff73"
server_key_version: "encryptionkey@0"
I20260925 15:13:18.093395 16495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349162370577-12320-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349162370577-12320-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349162370577-12320-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:13:18.124307 16495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:13:18.128558 16495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:13:18.128837 16495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:13:18.129771 16495 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:13:18.131139 16495 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:13:18.131268 16495 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:18.131400 16495 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:13:18.131489 16495 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:18.164987 16495 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.1:41157
I20260925 15:13:18.165076 16621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.1:41157 every 8 connection(s)
I20260925 15:13:18.165864 16495 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349162370577-12320-0/minicluster-data/ts-0/data/info.pb
I20260925 15:13:18.167308 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 16495
I20260925 15:13:18.167526 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349162370577-12320-0/minicluster-data/ts-0/wal/instance
I20260925 15:13:18.168183 12320 external_mini_cluster.cc:1428] Setting key 28a440387043ad044a6f3eb4ed34f7fe
Sep 25 15:13:18 dist-test-slave-p468 krb5kdc[16389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349198, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.1@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
I20260925 15:13:18.187500 16624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:18.168340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.1:42843 (local address 127.12.8.62:41241)
0925 15:13:18.168933 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:18.168942 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:18.170091 (+ 1149us) server_negotiation.cc:407] Connection header received
0925 15:13:18.171773 (+ 1682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:18.171778 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:18.171857 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:18.171988 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:18.173462 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:18.174245 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:18.176420 (+ 2175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:18.177177 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:18.177323 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:18.180648 (+ 3325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:18.180658 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:18.180665 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:18.180714 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:18.184320 (+ 3606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:18.185014 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:18.185023 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:18.185029 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:18.185182 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:18.185608 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:18.185613 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:18.185618 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:18.185968 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:18.186128 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:18.186829 (+ 701us) server_negotiation.cc:299] Negotiation successful
0925 15:13:18.187065 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":238,"threads_started":1}
WARNING: no policy specified for kudu/127.12.8.2@KRBTEST.COM; defaulting to no policy
I20260925 15:13:18.188134 16625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:18.169011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:41241 (local address 127.12.8.1:42843)
0925 15:13:18.169750 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:18.169807 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:18.171509 (+ 1702us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:18.172230 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:18.172238 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:18.172741 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:13:18.173298 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:18.173315 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:18.174400 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:18.174406 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:18.176270 (+ 1864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:18.176282 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:18.177386 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:18.177391 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:18.177584 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:18.178382 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:13:18.178408 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:13:18.180436 (+ 2028us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:13:18.184470 (+ 4034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:13:18.184477 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:13:18.184489 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:13:18.184878 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:13:18.185312 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:13:18.185315 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:13:18.185318 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:13:18.185495 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:13:18.186089 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:13:18.186094 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:13:18.186451 (+ 357us) client_negotiation.cc:769] Sending connection context
0925 15:13:18.186686 (+ 235us) client_negotiation.cc:240] Negotiation successful
0925 15:13:18.187036 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":267,"threads_started":1}
I20260925 15:13:18.189723 16622 heartbeater.cc:347] Connected to a master server at 127.12.8.62:41241
I20260925 15:13:18.190021 16622 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.2@KRBTEST.COM" created.
I20260925 15:13:18.190828 16622 heartbeater.cc:511] Master 127.12.8.62:41241 requested a full tablet report, sending...
I20260925 15:13:18.193077 16433 ts_manager.cc:194] Registered new tserver with Master: 6bc48e92ec99412f877fd1a4d5770edd (127.12.8.1:41157)
I20260925 15:13:18.195331 16433 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.1@KRBTEST.COM'} at 127.12.8.1:42843
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.2.keytab.
Entry for principal kudu/127.12.8.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.2.keytab.
I20260925 15:13:18.215677 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349162370577-12320-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349162370577-12320-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349162370577-12320-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349162370577-12320-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.2:0
--local_ip_for_outbound_sockets=127.12.8.2
--webserver_interface=127.12.8.2
--webserver_port=0
--tserver_master_addrs=127.12.8.62:41241
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.2.keytab
--principal=kudu/127.12.8.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:46795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:13:18.525233 16628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:13:18.525489 16628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:13:18.525648 16628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:13:18.532541 16628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:13:18.532631 16628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:13:18.532770 16628 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.2
I20260925 15:13:18.541996 16628 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:46795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349162370577-12320-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349162370577-12320-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.2.keytab
--principal=kudu/127.12.8.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349162370577-12320-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.8.2
--webserver_port=0
--tserver_master_addrs=127.12.8.62:41241
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.2
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349162370577-12320-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:13:18.543185 16628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:13:18.544406 16628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:13:18.556945 16634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:18.556967 16636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:18.557554 16633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:13:18.557885 16628 server_base.cc:1034] running on GCE node
I20260925 15:13:18.558708 16628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:13:18.559582 16628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:13:18.560796 16628 hybrid_clock.cc:648] HybridClock initialized: now 1790349198560753 us; error 38 us; skew 500 ppm
I20260925 15:13:18.561092 16628 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:13:18 dist-test-slave-p468 krb5kdc[16389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349198, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:13:18.564204 16628 init.cc:377] Logged in from keytab as kudu/127.12.8.2@KRBTEST.COM (short username kudu)
I20260925 15:13:18.565332 16628 fs_manager.cc:359] Metadata directory not provided
I20260925 15:13:18.565467 16628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:13:18.565757 16628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:13:18.568495 16628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349162370577-12320-0/minicluster-data/ts-1/data/instance:
uuid: "bc050f5ccbed45be99d513beee93a725"
format_stamp: "Formatted at 2026-09-25 15:13:18 on dist-test-slave-p468"
server_key: "a378daf8e44ea9c29e4433f9748bb6e1"
server_key_iv: "7e4c5ecec672ae92f738ee0ea3683ad0"
server_key_version: "encryptionkey@0"
I20260925 15:13:18.569263 16628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349162370577-12320-0/minicluster-data/ts-1/wal/instance:
uuid: "bc050f5ccbed45be99d513beee93a725"
format_stamp: "Formatted at 2026-09-25 15:13:18 on dist-test-slave-p468"
server_key: "a378daf8e44ea9c29e4433f9748bb6e1"
server_key_iv: "7e4c5ecec672ae92f738ee0ea3683ad0"
server_key_version: "encryptionkey@0"
I20260925 15:13:18.573906 16628 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260925 15:13:18.577246 16642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:18.578574 16628 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:13:18.578737 16628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349162370577-12320-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349162370577-12320-0/minicluster-data/ts-1/wal
uuid: "bc050f5ccbed45be99d513beee93a725"
format_stamp: "Formatted at 2026-09-25 15:13:18 on dist-test-slave-p468"
server_key: "a378daf8e44ea9c29e4433f9748bb6e1"
server_key_iv: "7e4c5ecec672ae92f738ee0ea3683ad0"
server_key_version: "encryptionkey@0"
I20260925 15:13:18.578943 16628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349162370577-12320-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349162370577-12320-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349162370577-12320-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:13:18.595891 16628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:13:18.599948 16628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:13:18.600209 16628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:13:18.601107 16628 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:13:18.602404 16628 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:13:18.602520 16628 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:18.602648 16628 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:13:18.602733 16628 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:18.634654 16628 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.2:46111
I20260925 15:13:18.634737 16754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.2:46111 every 8 connection(s)
I20260925 15:13:18.635507 16628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349162370577-12320-0/minicluster-data/ts-1/data/info.pb
I20260925 15:13:18.643224 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 16628
I20260925 15:13:18.643430 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349162370577-12320-0/minicluster-data/ts-1/wal/instance
I20260925 15:13:18.643935 12320 external_mini_cluster.cc:1428] Setting key 8952f0d2ce6483e8b46e19d35ea19ccb
Sep 25 15:13:18 dist-test-slave-p468 krb5kdc[16389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349198, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.2@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
I20260925 15:13:18.656714 16624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:18.638109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.2:35979 (local address 127.12.8.62:41241)
0925 15:13:18.638344 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:18.638349 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:18.639501 (+ 1152us) server_negotiation.cc:407] Connection header received
0925 15:13:18.641208 (+ 1707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:18.641214 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:18.641301 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:18.641411 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:18.642899 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:18.643714 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:18.645653 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:18.646361 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:18.646510 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:18.649963 (+ 3453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:18.649973 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:18.649982 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:18.650031 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:18.653337 (+ 3306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:18.654302 (+ 965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:18.654306 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:18.654309 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:18.654439 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:18.654935 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:18.654940 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:18.654942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:18.655206 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:18.655340 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:18.656254 (+ 914us) server_negotiation.cc:299] Negotiation successful
0925 15:13:18.656454 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260925 15:13:18.658037 16757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:18.638641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:41241 (local address 127.12.8.2:35979)
0925 15:13:18.639321 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:18.639382 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:18.640947 (+ 1565us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:18.641624 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:18.641632 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:18.642114 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:13:18.642712 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:18.642729 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:18.643854 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:18.643860 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:18.645493 (+ 1633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:18.645506 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:18.646560 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:18.646565 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:18.646740 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:18.647396 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:13:18.647417 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:13:18.649731 (+ 2314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:13:18.653468 (+ 3737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:13:18.653479 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:13:18.653498 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:13:18.654115 (+ 617us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:13:18.654552 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:13:18.654559 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:13:18.654566 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:13:18.654786 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:13:18.655347 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:13:18.655357 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:13:18.655778 (+ 421us) client_negotiation.cc:769] Sending connection context
0925 15:13:18.656070 (+ 292us) client_negotiation.cc:240] Negotiation successful
0925 15:13:18.656633 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":277,"threads_started":1}
I20260925 15:13:18.659546 16755 heartbeater.cc:347] Connected to a master server at 127.12.8.62:41241
I20260925 15:13:18.659885 16755 heartbeater.cc:464] Registering TS with master...
I20260925 15:13:18.660550 16755 heartbeater.cc:511] Master 127.12.8.62:41241 requested a full tablet report, sending...
I20260925 15:13:18.662036 16433 ts_manager.cc:194] Registered new tserver with Master: bc050f5ccbed45be99d513beee93a725 (127.12.8.2:46111)
I20260925 15:13:18.662894 16433 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.2@KRBTEST.COM'} at 127.12.8.2:35979
WARNING: no policy specified for kudu/127.12.8.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.3.keytab.
Entry for principal kudu/127.12.8.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.3.keytab.
I20260925 15:13:18.689033 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349162370577-12320-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349162370577-12320-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349162370577-12320-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349162370577-12320-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.3:0
--local_ip_for_outbound_sockets=127.12.8.3
--webserver_interface=127.12.8.3
--webserver_port=0
--tserver_master_addrs=127.12.8.62:41241
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.3.keytab
--principal=kudu/127.12.8.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:46795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:13:19.000093 16760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:13:19.000357 16760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:13:19.000451 16760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:13:19.007812 16760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:13:19.007926 16760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:13:19.008042 16760 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.3
I20260925 15:13:19.017532 16760 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:46795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349162370577-12320-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349162370577-12320-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.3.keytab
--principal=kudu/127.12.8.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349162370577-12320-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.8.3
--webserver_port=0
--tserver_master_addrs=127.12.8.62:41241
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.3
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349162370577-12320-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:13:19.018781 16760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:13:19.020061 16760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:13:19.032366 16768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:19.032379 16766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:19.032526 16765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:13:19.033075 16760 server_base.cc:1034] running on GCE node
I20260925 15:13:19.033634 16760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:13:19.034595 16760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:13:19.035826 16760 hybrid_clock.cc:648] HybridClock initialized: now 1790349199035778 us; error 41 us; skew 500 ppm
I20260925 15:13:19.036131 16760 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:13:19 dist-test-slave-p468 krb5kdc[16389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349199, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:13:19.039206 16760 init.cc:377] Logged in from keytab as kudu/127.12.8.3@KRBTEST.COM (short username kudu)
I20260925 15:13:19.040331 16760 fs_manager.cc:359] Metadata directory not provided
I20260925 15:13:19.040452 16760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:13:19.040764 16760 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:13:19.043517 16760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349162370577-12320-0/minicluster-data/ts-2/data/instance:
uuid: "2794e7c2cfd54d78a7e0bfd542bd883c"
format_stamp: "Formatted at 2026-09-25 15:13:19 on dist-test-slave-p468"
server_key: "ecf1aec5669d9580724126878acc2526"
server_key_iv: "9267a4016e298ca8b302da7feeef9a88"
server_key_version: "encryptionkey@0"
I20260925 15:13:19.044216 16760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349162370577-12320-0/minicluster-data/ts-2/wal/instance:
uuid: "2794e7c2cfd54d78a7e0bfd542bd883c"
format_stamp: "Formatted at 2026-09-25 15:13:19 on dist-test-slave-p468"
server_key: "ecf1aec5669d9580724126878acc2526"
server_key_iv: "9267a4016e298ca8b302da7feeef9a88"
server_key_version: "encryptionkey@0"
I20260925 15:13:19.048777 16760 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 15:13:19.051890 16774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:19.053092 16760 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 15:13:19.053220 16760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349162370577-12320-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349162370577-12320-0/minicluster-data/ts-2/wal
uuid: "2794e7c2cfd54d78a7e0bfd542bd883c"
format_stamp: "Formatted at 2026-09-25 15:13:19 on dist-test-slave-p468"
server_key: "ecf1aec5669d9580724126878acc2526"
server_key_iv: "9267a4016e298ca8b302da7feeef9a88"
server_key_version: "encryptionkey@0"
I20260925 15:13:19.053411 16760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349162370577-12320-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349162370577-12320-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349162370577-12320-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:13:19.073083 16760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:13:19.077288 16760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:13:19.077499 16760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:13:19.078383 16760 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:13:19.079586 16760 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:13:19.079667 16760 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:19.079751 16760 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:13:19.079804 16760 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:19.113618 16760 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.3:40053
I20260925 15:13:19.113694 16886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.3:40053 every 8 connection(s)
I20260925 15:13:19.114555 16760 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349162370577-12320-0/minicluster-data/ts-2/data/info.pb
I20260925 15:13:19.115984 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 16760
I20260925 15:13:19.116192 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349162370577-12320-0/minicluster-data/ts-2/wal/instance
I20260925 15:13:19.116627 12320 external_mini_cluster.cc:1428] Setting key c6db84ef4cb7bfaa586b0cada0e60f0c
Sep 25 15:13:19 dist-test-slave-p468 krb5kdc[16389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349199, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.3@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
I20260925 15:13:19.133634 16624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:19.117170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.3:38229 (local address 127.12.8.62:41241)
0925 15:13:19.117372 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:19.117378 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:19.118633 (+ 1255us) server_negotiation.cc:407] Connection header received
0925 15:13:19.120500 (+ 1867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:19.120504 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:19.120564 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:19.120643 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:19.122009 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:19.122876 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:19.124310 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:19.124684 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:19.124776 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:19.127939 (+ 3163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:19.127944 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:19.127947 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:19.127977 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:19.131182 (+ 3205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:19.131866 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:19.131870 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:19.131872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:19.131979 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:19.132313 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:19.132316 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:19.132318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:19.132551 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:19.132660 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:19.133235 (+ 575us) server_negotiation.cc:299] Negotiation successful
0925 15:13:19.133392 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 15:13:19.134559 16889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:19.117692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:41241 (local address 127.12.8.3:38229)
0925 15:13:19.118505 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:19.118550 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:19.120258 (+ 1708us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:19.120831 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:19.120839 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:19.121316 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:13:19.121853 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:19.121870 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:19.122997 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:19.123001 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:19.124180 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:19.124191 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:19.124864 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:19.124869 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:19.125051 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:19.125659 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:13:19.125679 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:13:19.127743 (+ 2064us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:13:19.131296 (+ 3553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:13:19.131302 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:13:19.131313 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:13:19.131766 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:13:19.132069 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:13:19.132072 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:13:19.132074 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:13:19.132224 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:13:19.132661 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:13:19.132666 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:13:19.132927 (+ 261us) client_negotiation.cc:769] Sending connection context
0925 15:13:19.133134 (+ 207us) client_negotiation.cc:240] Negotiation successful
0925 15:13:19.133482 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":338,"threads_started":1}
I20260925 15:13:19.135888 16887 heartbeater.cc:347] Connected to a master server at 127.12.8.62:41241
I20260925 15:13:19.136163 16887 heartbeater.cc:464] Registering TS with master...
I20260925 15:13:19.136770 16887 heartbeater.cc:511] Master 127.12.8.62:41241 requested a full tablet report, sending...
I20260925 15:13:19.138094 16433 ts_manager.cc:194] Registered new tserver with Master: 2794e7c2cfd54d78a7e0bfd542bd883c (127.12.8.3:40053)
I20260925 15:13:19.138880 16433 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.3@KRBTEST.COM'} at 127.12.8.3:38229
I20260925 15:13:19.143567 12320 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:13:19.159693 16624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:19.147889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35348 (local address 127.12.8.62:41241)
0925 15:13:19.148098 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:19.148106 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:19.148489 (+ 383us) server_negotiation.cc:407] Connection header received
0925 15:13:19.148613 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:19.148617 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:19.148669 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:19.148745 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:19.150042 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:19.150746 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:19.151902 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:19.152316 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:19.152422 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:19.154504 (+ 2082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:19.154509 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:19.154512 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:19.154543 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:19.157056 (+ 2513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:19.157896 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:19.157901 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:19.157903 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:19.158012 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:19.158373 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:19.158377 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:19.158378 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:19.158654 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:19.158829 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:19.159207 (+ 378us) server_negotiation.cc:299] Negotiation successful
0925 15:13:19.159452 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 15:13:19.163555 16433 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35348:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:13:19.170112 16433 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:13:19.190526 16900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:19.181254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32972 (local address 127.12.8.2:46111)
0925 15:13:19.181782 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:19.181791 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:19.182059 (+ 268us) server_negotiation.cc:407] Connection header received
0925 15:13:19.182247 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:19.182258 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:19.182517 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:19.182707 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:13:19.183847 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:19.185969 (+ 2122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:19.188494 (+ 2525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:19.189373 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:19.189520 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:19.189720 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:19.190014 (+ 294us) server_negotiation.cc:299] Negotiation successful
0925 15:13:19.190173 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":541,"threads_started":1}
I20260925 15:13:19.190557 16899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:19.180859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.2:46111 (local address 127.0.0.1:32972)
0925 15:13:19.181788 (+ 929us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:19.181833 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:19.182006 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:19.182988 (+ 982us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:19.182996 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:19.183032 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:13:19.183619 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:19.183637 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:19.186097 (+ 2460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:19.186102 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:19.188309 (+ 2207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:19.188334 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:19.189570 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:19.189577 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:19.189796 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:19.189847 (+ 51us) client_negotiation.cc:769] Sending connection context
0925 15:13:19.189976 (+ 129us) client_negotiation.cc:240] Negotiation successful
0925 15:13:19.190150 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":699,"mutex_wait_us":5,"thread_start_us":544,"threads_started":1}
I20260925 15:13:19.192698 16901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:19.182143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.3:40053 (local address 127.0.0.1:55422)
0925 15:13:19.182767 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:19.182788 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:19.182931 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:19.185116 (+ 2185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:19.185122 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:19.185145 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:13:19.185538 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:19.185553 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:19.187509 (+ 1956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:19.187514 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:19.189696 (+ 2182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:19.189704 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:19.191733 (+ 2029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:19.191741 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:19.191968 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:19.192001 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 15:13:19.192076 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 15:13:19.192189 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"mutex_wait_us":27,"thread_start_us":283,"threads_started":1}
I20260925 15:13:19.192740 16903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:19.182735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.1:41157 (local address 127.0.0.1:37496)
0925 15:13:19.183531 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:19.183554 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:19.183751 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:19.185375 (+ 1624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:19.185378 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:19.185400 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:13:19.185771 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:19.185783 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:19.187160 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:19.187165 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:19.189669 (+ 2504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:19.189690 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:19.191555 (+ 1865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:19.191561 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:19.192048 (+ 487us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:19.192085 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 15:13:19.192174 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 15:13:19.192293 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"mutex_wait_us":112,"thread_start_us":551,"threads_started":1}
I20260925 15:13:19.193123 16904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:19.183155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37496 (local address 127.12.8.1:41157)
0925 15:13:19.184620 (+ 1465us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:19.184626 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:19.184649 (+ 23us) server_negotiation.cc:407] Connection header received
0925 15:13:19.184724 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:19.184730 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:19.184930 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:19.185114 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:13:19.185946 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:19.187021 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:19.189862 (+ 2841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:19.191353 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:19.191487 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:19.191721 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:19.192640 (+ 919us) server_negotiation.cc:299] Negotiation successful
0925 15:13:19.192814 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1313,"thread_start_us":1167,"threads_started":1}
I20260925 15:13:19.193691 16902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:19.182433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55422 (local address 127.12.8.3:40053)
0925 15:13:19.184581 (+ 2148us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:19.184591 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:19.184623 (+ 32us) server_negotiation.cc:407] Connection header received
0925 15:13:19.184718 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:19.184724 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:19.184927 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:19.185118 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:13:19.185753 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:19.187371 (+ 1618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:19.189862 (+ 2491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:19.191535 (+ 1673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:19.192941 (+ 1406us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:19.193156 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:19.193238 (+ 82us) server_negotiation.cc:299] Negotiation successful
0925 15:13:19.193390 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1986,"thread_start_us":1568,"threads_started":1}
I20260925 15:13:19.194312 16690 tablet_service.cc:1467] Processing CreateTablet for tablet 6113923643e844ad8c4e92c8d491f670 (DEFAULT_TABLE table=test-table [id=41ddf9e808504726860ca67511535768]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:13:19.195808 16690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6113923643e844ad8c4e92c8d491f670. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:13:19.196228 16557 tablet_service.cc:1467] Processing CreateTablet for tablet 6113923643e844ad8c4e92c8d491f670 (DEFAULT_TABLE table=test-table [id=41ddf9e808504726860ca67511535768]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:13:19.197494 16557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6113923643e844ad8c4e92c8d491f670. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:13:19.197118 16822 tablet_service.cc:1467] Processing CreateTablet for tablet 6113923643e844ad8c4e92c8d491f670 (DEFAULT_TABLE table=test-table [id=41ddf9e808504726860ca67511535768]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:13:19.198383 16822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6113923643e844ad8c4e92c8d491f670. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:13:19.198678 16622 heartbeater.cc:503] Master 127.12.8.62:41241 was elected leader, sending a full tablet report...
I20260925 15:13:19.209288 16905 tablet_bootstrap.cc:492] T 6113923643e844ad8c4e92c8d491f670 P 6bc48e92ec99412f877fd1a4d5770edd: Bootstrap starting.
I20260925 15:13:19.212347 16907 tablet_bootstrap.cc:492] T 6113923643e844ad8c4e92c8d491f670 P 2794e7c2cfd54d78a7e0bfd542bd883c: Bootstrap starting.
I20260925 15:13:19.213701 16906 tablet_bootstrap.cc:492] T 6113923643e844ad8c4e92c8d491f670 P bc050f5ccbed45be99d513beee93a725: Bootstrap starting.
I20260925 15:13:19.213805 16905 tablet_bootstrap.cc:654] T 6113923643e844ad8c4e92c8d491f670 P 6bc48e92ec99412f877fd1a4d5770edd: Neither blocks nor log segments found. Creating new log.
I20260925 15:13:19.215197 16905 log.cc:824] T 6113923643e844ad8c4e92c8d491f670 P 6bc48e92ec99412f877fd1a4d5770edd: Log is configured to *not* fsync() on all Append() calls
I20260925 15:13:19.216039 16907 tablet_bootstrap.cc:654] T 6113923643e844ad8c4e92c8d491f670 P 2794e7c2cfd54d78a7e0bfd542bd883c: Neither blocks nor log segments found. Creating new log.
I20260925 15:13:19.216679 16906 tablet_bootstrap.cc:654] T 6113923643e844ad8c4e92c8d491f670 P bc050f5ccbed45be99d513beee93a725: Neither blocks nor log segments found. Creating new log.
I20260925 15:13:19.217191 16907 log.cc:824] T 6113923643e844ad8c4e92c8d491f670 P 2794e7c2cfd54d78a7e0bfd542bd883c: Log is configured to *not* fsync() on all Append() calls
I20260925 15:13:19.217610 16906 log.cc:824] T 6113923643e844ad8c4e92c8d491f670 P bc050f5ccbed45be99d513beee93a725: Log is configured to *not* fsync() on all Append() calls
I20260925 15:13:19.218920 16905 tablet_bootstrap.cc:492] T 6113923643e844ad8c4e92c8d491f670 P 6bc48e92ec99412f877fd1a4d5770edd: No bootstrap required, opened a new log
I20260925 15:13:19.219234 16905 ts_tablet_manager.cc:1397] T 6113923643e844ad8c4e92c8d491f670 P 6bc48e92ec99412f877fd1a4d5770edd: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.005s
I20260925 15:13:19.220012 16907 tablet_bootstrap.cc:492] T 6113923643e844ad8c4e92c8d491f670 P 2794e7c2cfd54d78a7e0bfd542bd883c: No bootstrap required, opened a new log
I20260925 15:13:19.220257 16907 ts_tablet_manager.cc:1397] T 6113923643e844ad8c4e92c8d491f670 P 2794e7c2cfd54d78a7e0bfd542bd883c: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260925 15:13:19.220446 16906 tablet_bootstrap.cc:492] T 6113923643e844ad8c4e92c8d491f670 P bc050f5ccbed45be99d513beee93a725: No bootstrap required, opened a new log
I20260925 15:13:19.220680 16906 ts_tablet_manager.cc:1397] T 6113923643e844ad8c4e92c8d491f670 P bc050f5ccbed45be99d513beee93a725: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260925 15:13:19.227255 16907 raft_consensus.cc:348] T 6113923643e844ad8c4e92c8d491f670 P 2794e7c2cfd54d78a7e0bfd542bd883c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc050f5ccbed45be99d513beee93a725" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 46111 } } peers { permanent_uuid: "2794e7c2cfd54d78a7e0bfd542bd883c" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 40053 } } peers { permanent_uuid: "6bc48e92ec99412f877fd1a4d5770edd" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 41157 } }
I20260925 15:13:19.227613 16907 raft_consensus.cc:374] T 6113923643e844ad8c4e92c8d491f670 P 2794e7c2cfd54d78a7e0bfd542bd883c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:13:19.227713 16907 raft_consensus.cc:729] T 6113923643e844ad8c4e92c8d491f670 P 2794e7c2cfd54d78a7e0bfd542bd883c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2794e7c2cfd54d78a7e0bfd542bd883c, State: Initialized, Role: FOLLOWER
I20260925 15:13:19.228327 16907 consensus_queue.cc:260] T 6113923643e844ad8c4e92c8d491f670 P 2794e7c2cfd54d78a7e0bfd542bd883c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc050f5ccbed45be99d513beee93a725" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 46111 } } peers { permanent_uuid: "2794e7c2cfd54d78a7e0bfd542bd883c" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 40053 } } peers { permanent_uuid: "6bc48e92ec99412f877fd1a4d5770edd" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 41157 } }
I20260925 15:13:19.229375 16887 heartbeater.cc:503] Master 127.12.8.62:41241 was elected leader, sending a full tablet report...
I20260925 15:13:19.230036 16907 ts_tablet_manager.cc:1428] T 6113923643e844ad8c4e92c8d491f670 P 2794e7c2cfd54d78a7e0bfd542bd883c: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260925 15:13:19.230779 16906 raft_consensus.cc:348] T 6113923643e844ad8c4e92c8d491f670 P bc050f5ccbed45be99d513beee93a725 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc050f5ccbed45be99d513beee93a725" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 46111 } } peers { permanent_uuid: "2794e7c2cfd54d78a7e0bfd542bd883c" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 40053 } } peers { permanent_uuid: "6bc48e92ec99412f877fd1a4d5770edd" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 41157 } }
I20260925 15:13:19.231199 16906 raft_consensus.cc:374] T 6113923643e844ad8c4e92c8d491f670 P bc050f5ccbed45be99d513beee93a725 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:13:19.230980 16905 raft_consensus.cc:348] T 6113923643e844ad8c4e92c8d491f670 P 6bc48e92ec99412f877fd1a4d5770edd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc050f5ccbed45be99d513beee93a725" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 46111 } } peers { permanent_uuid: "2794e7c2cfd54d78a7e0bfd542bd883c" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 40053 } } peers { permanent_uuid: "6bc48e92ec99412f877fd1a4d5770edd" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 41157 } }
I20260925 15:13:19.231302 16906 raft_consensus.cc:729] T 6113923643e844ad8c4e92c8d491f670 P bc050f5ccbed45be99d513beee93a725 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc050f5ccbed45be99d513beee93a725, State: Initialized, Role: FOLLOWER
I20260925 15:13:19.231341 16905 raft_consensus.cc:374] T 6113923643e844ad8c4e92c8d491f670 P 6bc48e92ec99412f877fd1a4d5770edd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:13:19.231428 16905 raft_consensus.cc:729] T 6113923643e844ad8c4e92c8d491f670 P 6bc48e92ec99412f877fd1a4d5770edd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bc48e92ec99412f877fd1a4d5770edd, State: Initialized, Role: FOLLOWER
I20260925 15:13:19.231910 16906 consensus_queue.cc:260] T 6113923643e844ad8c4e92c8d491f670 P bc050f5ccbed45be99d513beee93a725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc050f5ccbed45be99d513beee93a725" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 46111 } } peers { permanent_uuid: "2794e7c2cfd54d78a7e0bfd542bd883c" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 40053 } } peers { permanent_uuid: "6bc48e92ec99412f877fd1a4d5770edd" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 41157 } }
I20260925 15:13:19.232023 16905 consensus_queue.cc:260] T 6113923643e844ad8c4e92c8d491f670 P 6bc48e92ec99412f877fd1a4d5770edd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc050f5ccbed45be99d513beee93a725" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 46111 } } peers { permanent_uuid: "2794e7c2cfd54d78a7e0bfd542bd883c" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 40053 } } peers { permanent_uuid: "6bc48e92ec99412f877fd1a4d5770edd" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 41157 } }
I20260925 15:13:19.233016 16755 heartbeater.cc:503] Master 127.12.8.62:41241 was elected leader, sending a full tablet report...
I20260925 15:13:19.233872 16906 ts_tablet_manager.cc:1428] T 6113923643e844ad8c4e92c8d491f670 P bc050f5ccbed45be99d513beee93a725: Time spent starting tablet: real 0.013s user 0.011s sys 0.003s
I20260925 15:13:19.237504 16905 ts_tablet_manager.cc:1428] T 6113923643e844ad8c4e92c8d491f670 P 6bc48e92ec99412f877fd1a4d5770edd: Time spent starting tablet: real 0.018s user 0.002s sys 0.013s
W20260925 15:13:19.366070 16888 tablet.cc:2369] T 6113923643e844ad8c4e92c8d491f670 P 2794e7c2cfd54d78a7e0bfd542bd883c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:13:19.387373 16756 tablet.cc:2369] T 6113923643e844ad8c4e92c8d491f670 P bc050f5ccbed45be99d513beee93a725: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:13:19.418174 16623 tablet.cc:2369] T 6113923643e844ad8c4e92c8d491f670 P 6bc48e92ec99412f877fd1a4d5770edd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:13:19.454700 16912 raft_consensus.cc:482] T 6113923643e844ad8c4e92c8d491f670 P bc050f5ccbed45be99d513beee93a725 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:13:19.455000 16912 raft_consensus.cc:504] T 6113923643e844ad8c4e92c8d491f670 P bc050f5ccbed45be99d513beee93a725 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc050f5ccbed45be99d513beee93a725" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 46111 } } peers { permanent_uuid: "2794e7c2cfd54d78a7e0bfd542bd883c" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 40053 } } peers { permanent_uuid: "6bc48e92ec99412f877fd1a4d5770edd" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 41157 } }
I20260925 15:13:19.456676 16912 leader_election.cc:288] T 6113923643e844ad8c4e92c8d491f670 P bc050f5ccbed45be99d513beee93a725 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2794e7c2cfd54d78a7e0bfd542bd883c (127.12.8.3:40053), 6bc48e92ec99412f877fd1a4d5770edd (127.12.8.1:41157)
I20260925 15:13:19.463366 16902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:19.457201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.2:51057 (local address 127.12.8.3:40053)
0925 15:13:19.457439 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:19.457443 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:19.458082 (+ 639us) server_negotiation.cc:407] Connection header received
0925 15:13:19.458347 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:19.458354 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:19.458430 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:19.458496 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:13:19.459168 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:19.459886 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:19.461629 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:19.462478 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:19.462622 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:19.462730 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:19.462997 (+ 267us) server_negotiation.cc:299] Negotiation successful
0925 15:13:19.463118 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 15:13:19.463550 16914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:19.457135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.3:40053 (local address 127.12.8.2:51057)
0925 15:13:19.457927 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:19.457946 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:19.458087 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:19.458668 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:19.458672 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:19.458693 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:13:19.459021 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:19.459029 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:19.460015 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:19.460023 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:19.461473 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:19.461484 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:19.462682 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:19.462687 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:19.462837 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:19.462860 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 15:13:19.462971 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 15:13:19.463102 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"mutex_wait_us":1,"spinlock_wait_cycles":18816,"thread_start_us":431,"threads_started":1}
I20260925 15:13:19.463747 16904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:19.458248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.2:45659 (local address 127.12.8.1:41157)
0925 15:13:19.458450 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:19.458458 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:19.458705 (+ 247us) server_negotiation.cc:407] Connection header received
0925 15:13:19.458955 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:19.458959 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:19.459013 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:19.459104 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:13:19.459610 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:19.460319 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:19.462017 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:19.462832 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:19.462988 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:19.463088 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:19.463379 (+ 291us) server_negotiation.cc:299] Negotiation successful
0925 15:13:19.463499 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 15:13:19.463912 16915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:19.457995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.1:41157 (local address 127.12.8.2:45659)
0925 15:13:19.458594 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:19.458607 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:19.458765 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:19.459116 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:19.459120 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:19.459142 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:13:19.459464 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:19.459472 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:19.460453 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:19.460459 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:19.461859 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:19.461870 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:19.463048 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:19.463054 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:19.463214 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:19.463240 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 15:13:19.463361 (+ 121us) client_negotiation.cc:240] Negotiation successful
0925 15:13:19.463483 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"mutex_wait_us":20,"thread_start_us":322,"threads_started":1}
I20260925 15:13:19.464658 16842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6113923643e844ad8c4e92c8d491f670" candidate_uuid: "bc050f5ccbed45be99d513beee93a725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2794e7c2cfd54d78a7e0bfd542bd883c" is_pre_election: true
I20260925 15:13:19.465041 16577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6113923643e844ad8c4e92c8d491f670" candidate_uuid: "bc050f5ccbed45be99d513beee93a725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bc48e92ec99412f877fd1a4d5770edd" is_pre_election: true
I20260925 15:13:19.465222 16842 raft_consensus.cc:2454] T 6113923643e844ad8c4e92c8d491f670 P 2794e7c2cfd54d78a7e0bfd542bd883c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc050f5ccbed45be99d513beee93a725 in term 0.
I20260925 15:13:19.465526 16577 raft_consensus.cc:2454] T 6113923643e844ad8c4e92c8d491f670 P 6bc48e92ec99412f877fd1a4d5770edd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc050f5ccbed45be99d513beee93a725 in term 0.
I20260925 15:13:19.466027 16646 leader_election.cc:302] T 6113923643e844ad8c4e92c8d491f670 P bc050f5ccbed45be99d513beee93a725 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2794e7c2cfd54d78a7e0bfd542bd883c, bc050f5ccbed45be99d513beee93a725; no voters:
I20260925 15:13:19.466500 16912 raft_consensus.cc:2781] T 6113923643e844ad8c4e92c8d491f670 P bc050f5ccbed45be99d513beee93a725 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:13:19.466667 16912 raft_consensus.cc:482] T 6113923643e844ad8c4e92c8d491f670 P bc050f5ccbed45be99d513beee93a725 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:13:19.466787 16912 raft_consensus.cc:3037] T 6113923643e844ad8c4e92c8d491f670 P bc050f5ccbed45be99d513beee93a725 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:13:19.469300 16912 raft_consensus.cc:504] T 6113923643e844ad8c4e92c8d491f670 P bc050f5ccbed45be99d513beee93a725 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc050f5ccbed45be99d513beee93a725" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 46111 } } peers { permanent_uuid: "2794e7c2cfd54d78a7e0bfd542bd883c" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 40053 } } peers { permanent_uuid: "6bc48e92ec99412f877fd1a4d5770edd" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 41157 } }
I20260925 15:13:19.469945 16912 leader_election.cc:288] T 6113923643e844ad8c4e92c8d491f670 P bc050f5ccbed45be99d513beee93a725 [CANDIDATE]: Term 1 election: Requested vote from peers 2794e7c2cfd54d78a7e0bfd542bd883c (127.12.8.3:40053), 6bc48e92ec99412f877fd1a4d5770edd (127.12.8.1:41157)
I20260925 15:13:19.470371 16842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6113923643e844ad8c4e92c8d491f670" candidate_uuid: "bc050f5ccbed45be99d513beee93a725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2794e7c2cfd54d78a7e0bfd542bd883c"
I20260925 15:13:19.470440 16577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6113923643e844ad8c4e92c8d491f670" candidate_uuid: "bc050f5ccbed45be99d513beee93a725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bc48e92ec99412f877fd1a4d5770edd"
I20260925 15:13:19.470573 16842 raft_consensus.cc:3037] T 6113923643e844ad8c4e92c8d491f670 P 2794e7c2cfd54d78a7e0bfd542bd883c [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:13:19.470680 16577 raft_consensus.cc:3037] T 6113923643e844ad8c4e92c8d491f670 P 6bc48e92ec99412f877fd1a4d5770edd [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:13:19.473148 16577 raft_consensus.cc:2454] T 6113923643e844ad8c4e92c8d491f670 P 6bc48e92ec99412f877fd1a4d5770edd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc050f5ccbed45be99d513beee93a725 in term 1.
I20260925 15:13:19.473296 16842 raft_consensus.cc:2454] T 6113923643e844ad8c4e92c8d491f670 P 2794e7c2cfd54d78a7e0bfd542bd883c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc050f5ccbed45be99d513beee93a725 in term 1.
I20260925 15:13:19.473572 16646 leader_election.cc:302] T 6113923643e844ad8c4e92c8d491f670 P bc050f5ccbed45be99d513beee93a725 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bc48e92ec99412f877fd1a4d5770edd, bc050f5ccbed45be99d513beee93a725; no voters:
I20260925 15:13:19.473894 16912 raft_consensus.cc:2781] T 6113923643e844ad8c4e92c8d491f670 P bc050f5ccbed45be99d513beee93a725 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:13:19.474402 16912 raft_consensus.cc:686] T 6113923643e844ad8c4e92c8d491f670 P bc050f5ccbed45be99d513beee93a725 [term 1 LEADER]: Becoming Leader. State: Replica: bc050f5ccbed45be99d513beee93a725, State: Running, Role: LEADER
I20260925 15:13:19.474977 16912 consensus_queue.cc:237] T 6113923643e844ad8c4e92c8d491f670 P bc050f5ccbed45be99d513beee93a725 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc050f5ccbed45be99d513beee93a725" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 46111 } } peers { permanent_uuid: "2794e7c2cfd54d78a7e0bfd542bd883c" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 40053 } } peers { permanent_uuid: "6bc48e92ec99412f877fd1a4d5770edd" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 41157 } }
I20260925 15:13:19.479365 16431 catalog_manager.cc:5495] T 6113923643e844ad8c4e92c8d491f670 P bc050f5ccbed45be99d513beee93a725 reported cstate change: term changed from 0 to 1, leader changed from <none> to bc050f5ccbed45be99d513beee93a725 (127.12.8.2). New cstate: current_term: 1 leader_uuid: "bc050f5ccbed45be99d513beee93a725" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc050f5ccbed45be99d513beee93a725" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 46111 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2794e7c2cfd54d78a7e0bfd542bd883c" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 40053 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6bc48e92ec99412f877fd1a4d5770edd" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 41157 } health_report { overall_health: UNKNOWN } } }
I20260925 15:13:19.554270 16900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:19.549248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32974 (local address 127.12.8.2:46111)
0925 15:13:19.549443 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:19.549447 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:19.549537 (+ 90us) server_negotiation.cc:407] Connection header received
0925 15:13:19.549698 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:19.549702 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:19.549785 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:19.549919 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:13:19.550251 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:19.550955 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:19.552208 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:19.552728 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:19.552822 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:19.553208 (+ 386us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:13:19.553725 (+ 517us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:13:19.553826 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:19.553954 (+ 128us) server_negotiation.cc:299] Negotiation successful
0925 15:13:19.554014 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 15:13:19.563042 16842 raft_consensus.cc:1261] T 6113923643e844ad8c4e92c8d491f670 P 2794e7c2cfd54d78a7e0bfd542bd883c [term 1 FOLLOWER]: Refusing update from remote peer bc050f5ccbed45be99d513beee93a725: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:13:19.563107 16577 raft_consensus.cc:1261] T 6113923643e844ad8c4e92c8d491f670 P 6bc48e92ec99412f877fd1a4d5770edd [term 1 FOLLOWER]: Refusing update from remote peer bc050f5ccbed45be99d513beee93a725: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:13:19.563925 16916 consensus_queue.cc:1035] T 6113923643e844ad8c4e92c8d491f670 P bc050f5ccbed45be99d513beee93a725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2794e7c2cfd54d78a7e0bfd542bd883c" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 40053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:13:19.564359 16912 consensus_queue.cc:1035] T 6113923643e844ad8c4e92c8d491f670 P bc050f5ccbed45be99d513beee93a725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bc48e92ec99412f877fd1a4d5770edd" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 41157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:13:19.591737 16432 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35348:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:13:19.592041 16432 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35348:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:13:19.595358 16432 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9dbb7b4f6abe47d091c220670710851c: Sending DeleteTablet for 3 replicas of tablet 6113923643e844ad8c4e92c8d491f670
I20260925 15:13:19.596438 16690 tablet_service.cc:1514] Processing DeleteTablet for tablet 6113923643e844ad8c4e92c8d491f670 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:13:19 UTC) from {username='kudu', principal='kudu/127.12.8.62@KRBTEST.COM'} at 127.0.0.1:32972
I20260925 15:13:19.596596 16822 tablet_service.cc:1514] Processing DeleteTablet for tablet 6113923643e844ad8c4e92c8d491f670 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:13:19 UTC) from {username='kudu', principal='kudu/127.12.8.62@KRBTEST.COM'} at 127.0.0.1:55422
I20260925 15:13:19.596757 16557 tablet_service.cc:1514] Processing DeleteTablet for tablet 6113923643e844ad8c4e92c8d491f670 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:13:19 UTC) from {username='kudu', principal='kudu/127.12.8.62@KRBTEST.COM'} at 127.0.0.1:37496
I20260925 15:13:19.599198 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 16495
I20260925 15:13:19.605399 16930 tablet_replica.cc:330] T 6113923643e844ad8c4e92c8d491f670 P 6bc48e92ec99412f877fd1a4d5770edd: stopping tablet replica
I20260925 15:13:19.606734 16930 raft_consensus.cc:2229] T 6113923643e844ad8c4e92c8d491f670 P 6bc48e92ec99412f877fd1a4d5770edd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:13:19.607824 16930 raft_consensus.cc:2258] T 6113923643e844ad8c4e92c8d491f670 P 6bc48e92ec99412f877fd1a4d5770edd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:13:19.610911 16930 ts_tablet_manager.cc:1905] T 6113923643e844ad8c4e92c8d491f670 P 6bc48e92ec99412f877fd1a4d5770edd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:13:19.616091 16928 tablet_replica.cc:330] T 6113923643e844ad8c4e92c8d491f670 P bc050f5ccbed45be99d513beee93a725: stopping tablet replica
I20260925 15:13:19.619684 16928 raft_consensus.cc:2229] T 6113923643e844ad8c4e92c8d491f670 P bc050f5ccbed45be99d513beee93a725 [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:13:19.620425 16928 raft_consensus.cc:2258] T 6113923643e844ad8c4e92c8d491f670 P bc050f5ccbed45be99d513beee93a725 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:13:19.621215 16930 ts_tablet_manager.cc:1918] T 6113923643e844ad8c4e92c8d491f670 P 6bc48e92ec99412f877fd1a4d5770edd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:13:19.621424 16930 log.cc:1196] T 6113923643e844ad8c4e92c8d491f670 P 6bc48e92ec99412f877fd1a4d5770edd: Deleting WAL directory at /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349162370577-12320-0/minicluster-data/ts-0/wal/wals/6113923643e844ad8c4e92c8d491f670
I20260925 15:13:19.622007 16930 ts_tablet_manager.cc:1939] T 6113923643e844ad8c4e92c8d491f670 P 6bc48e92ec99412f877fd1a4d5770edd: Deleting consensus metadata
I20260925 15:13:19.623817 16420 catalog_manager.cc:4841] TS 6bc48e92ec99412f877fd1a4d5770edd (127.12.8.1:41157): tablet 6113923643e844ad8c4e92c8d491f670 (table test-table [id=41ddf9e808504726860ca67511535768]) successfully deleted
I20260925 15:13:19.624378 16928 ts_tablet_manager.cc:1905] T 6113923643e844ad8c4e92c8d491f670 P bc050f5ccbed45be99d513beee93a725: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:13:19.625176 16929 tablet_replica.cc:330] T 6113923643e844ad8c4e92c8d491f670 P 2794e7c2cfd54d78a7e0bfd542bd883c: stopping tablet replica
I20260925 15:13:19.625566 16929 raft_consensus.cc:2229] T 6113923643e844ad8c4e92c8d491f670 P 2794e7c2cfd54d78a7e0bfd542bd883c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:13:19.626013 16929 raft_consensus.cc:2258] T 6113923643e844ad8c4e92c8d491f670 P 2794e7c2cfd54d78a7e0bfd542bd883c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:13:19.628062 16929 ts_tablet_manager.cc:1905] T 6113923643e844ad8c4e92c8d491f670 P 2794e7c2cfd54d78a7e0bfd542bd883c: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 25 15:13:19 dist-test-slave-p468 krb5kdc[16389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349197, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.1@KRBTEST.COM
I20260925 15:13:19.631752 16928 ts_tablet_manager.cc:1918] T 6113923643e844ad8c4e92c8d491f670 P bc050f5ccbed45be99d513beee93a725: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:13:19.631959 16928 log.cc:1196] T 6113923643e844ad8c4e92c8d491f670 P bc050f5ccbed45be99d513beee93a725: Deleting WAL directory at /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349162370577-12320-0/minicluster-data/ts-1/wal/wals/6113923643e844ad8c4e92c8d491f670
I20260925 15:13:19.632506 16928 ts_tablet_manager.cc:1939] T 6113923643e844ad8c4e92c8d491f670 P bc050f5ccbed45be99d513beee93a725: Deleting consensus metadata
I20260925 15:13:19.634155 16420 catalog_manager.cc:4841] TS bc050f5ccbed45be99d513beee93a725 (127.12.8.2:46111): tablet 6113923643e844ad8c4e92c8d491f670 (table test-table [id=41ddf9e808504726860ca67511535768]) successfully deleted
I20260925 15:13:19.634765 16929 ts_tablet_manager.cc:1918] T 6113923643e844ad8c4e92c8d491f670 P 2794e7c2cfd54d78a7e0bfd542bd883c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:13:19.634923 16929 log.cc:1196] T 6113923643e844ad8c4e92c8d491f670 P 2794e7c2cfd54d78a7e0bfd542bd883c: Deleting WAL directory at /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349162370577-12320-0/minicluster-data/ts-2/wal/wals/6113923643e844ad8c4e92c8d491f670
I20260925 15:13:19.635474 16929 ts_tablet_manager.cc:1939] T 6113923643e844ad8c4e92c8d491f670 P 2794e7c2cfd54d78a7e0bfd542bd883c: Deleting consensus metadata
I20260925 15:13:19.637096 16420 catalog_manager.cc:4841] TS 2794e7c2cfd54d78a7e0bfd542bd883c (127.12.8.3:40053): tablet 6113923643e844ad8c4e92c8d491f670 (table test-table [id=41ddf9e808504726860ca67511535768]) successfully deleted
I20260925 15:13:19.638139 16904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:19.600708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37508 (local address 127.12.8.1:41157)
0925 15:13:19.600930 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:19.600940 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:19.612968 (+ 12028us) server_negotiation.cc:407] Connection header received
0925 15:13:19.613405 (+ 437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:19.613416 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:19.613507 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:19.613574 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:19.616289 (+ 2715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:19.617445 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:19.621405 (+ 3960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:19.622115 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:19.622238 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:19.631046 (+ 8808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:19.631060 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:19.631076 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:19.631124 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:19.634892 (+ 3768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:19.635719 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:19.635728 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:19.635731 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:19.635913 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:19.636344 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:19.636349 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:19.636352 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:19.636775 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:19.637054 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:19.637556 (+ 502us) server_negotiation.cc:299] Negotiation successful
0925 15:13:19.637767 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 15:13:19.638790 16617 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:13:19.738111 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 16495
I20260925 15:13:19.755259 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 16628
Sep 25 15:13:19 dist-test-slave-p468 krb5kdc[16389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349197, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.2@KRBTEST.COM
I20260925 15:13:19.772176 16900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:19.756038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32982 (local address 127.12.8.2:46111)
0925 15:13:19.756235 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:19.756240 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:19.756310 (+ 70us) server_negotiation.cc:407] Connection header received
0925 15:13:19.756495 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:19.756499 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:19.756574 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:19.756655 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:19.758264 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:19.759100 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:19.760880 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:19.761293 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:19.761386 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:19.766361 (+ 4975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:19.766374 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:19.766391 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:19.766429 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:19.769421 (+ 2992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:19.770155 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:19.770161 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:19.770164 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:19.770395 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:19.770785 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:19.770790 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:19.770793 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:19.771114 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:19.771391 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:19.771672 (+ 281us) server_negotiation.cc:299] Negotiation successful
0925 15:13:19.771833 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260925 15:13:19.772977 16750 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:13:19.889637 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 16628
I20260925 15:13:19.907953 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 16760
Sep 25 15:13:19 dist-test-slave-p468 krb5kdc[16389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349197, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.3@KRBTEST.COM
I20260925 15:13:19.922722 16902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:19.908811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55436 (local address 127.12.8.3:40053)
0925 15:13:19.908987 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:19.908993 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:19.909010 (+ 17us) server_negotiation.cc:407] Connection header received
0925 15:13:19.909126 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:19.909130 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:19.909191 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:19.909335 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:19.910615 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:19.911359 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:19.912565 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:19.912972 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:19.913069 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:19.917564 (+ 4495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:19.917579 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:19.917595 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:19.917630 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:19.920430 (+ 2800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:19.921008 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:19.921015 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:19.921018 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:19.921142 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:19.921494 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:19.921497 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:19.921499 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:19.921832 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:19.922049 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:19.922324 (+ 275us) server_negotiation.cc:299] Negotiation successful
0925 15:13:19.922473 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 15:13:19.923466 16882 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:13:20.011490 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 16760
I20260925 15:13:20.027922 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 16402
I20260925 15:13:20.029089 16463 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:13:20.136771 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 16402
2026-09-25T15:13:20Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4258 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-25T15:13:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:13:20Z Disabled control of system clock
I20260925 15:13:20.200695 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349162370577-12320-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349162370577-12320-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349162370577-12320-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.62:43719
--webserver_interface=127.12.8.62
--webserver_port=0
--builtin_ntp_servers=127.12.8.20:37867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.62:43719
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 15:13:20.513314 16939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:13:20.513582 16939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:13:20.513739 16939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:13:20.520764 16939 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:13:20.520864 16939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:13:20.520953 16939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:13:20.521059 16939 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:13:20.521136 16939 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:13:20.529989 16939 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:37867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349162370577-12320-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349162370577-12320-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.8.62:43719
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.62:43719
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.8.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349162370577-12320-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:13:20.531167 16939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:13:20.532888 16939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:13:20.544361 16944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:20.544322 16945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:20.544409 16947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:13:20.545049 16939 server_base.cc:1034] running on GCE node
I20260925 15:13:20.545739 16939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:13:20.547109 16939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:13:20.548309 16939 hybrid_clock.cc:648] HybridClock initialized: now 1790349200548266 us; error 31 us; skew 500 ppm
I20260925 15:13:20.548642 16939 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:13:20.556653 16939 webserver.cc:460] Webserver started at http://127.12.8.62:34293/ using document root <none> and password file <none>
I20260925 15:13:20.557338 16939 fs_manager.cc:359] Metadata directory not provided
I20260925 15:13:20.557425 16939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:13:20.557766 16939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:13:20.560575 16939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349162370577-12320-0/minicluster-data/master-0/data/instance:
uuid: "02c7358a95cb4757bedee7c5342322af"
format_stamp: "Formatted at 2026-09-25 15:13:20 on dist-test-slave-p468"
server_key: "68f7ff5eb1ac73dd455d928dd7e459b8"
server_key_iv: "d8c70c504c22ba2258b3d4f8286588b4"
server_key_version: "encryptionkey@0"
I20260925 15:13:20.561359 16939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349162370577-12320-0/minicluster-data/master-0/wal/instance:
uuid: "02c7358a95cb4757bedee7c5342322af"
format_stamp: "Formatted at 2026-09-25 15:13:20 on dist-test-slave-p468"
server_key: "68f7ff5eb1ac73dd455d928dd7e459b8"
server_key_iv: "d8c70c504c22ba2258b3d4f8286588b4"
server_key_version: "encryptionkey@0"
I20260925 15:13:20.566246 16939 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260925 15:13:20.569386 16953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:20.570726 16939 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:13:20.570914 16939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349162370577-12320-0/minicluster-data/master-0/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349162370577-12320-0/minicluster-data/master-0/wal
uuid: "02c7358a95cb4757bedee7c5342322af"
format_stamp: "Formatted at 2026-09-25 15:13:20 on dist-test-slave-p468"
server_key: "68f7ff5eb1ac73dd455d928dd7e459b8"
server_key_iv: "d8c70c504c22ba2258b3d4f8286588b4"
server_key_version: "encryptionkey@0"
I20260925 15:13:20.571116 16939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349162370577-12320-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349162370577-12320-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349162370577-12320-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:13:20.576761 16939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:13:20.577589 16939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:13:20.577886 16939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:13:20.596709 16939 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.62:43719
I20260925 15:13:20.596751 17004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.62:43719 every 8 connection(s)
I20260925 15:13:20.598045 16939 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
I20260925 15:13:20.601824 17005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:13:20.606822 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 16939
I20260925 15:13:20.607012 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349162370577-12320-0/minicluster-data/master-0/wal/instance
I20260925 15:13:20.607762 12320 external_mini_cluster.cc:1428] Setting key 42ddd5749b8659f76f77b8a7fdce7392
I20260925 15:13:20.613914 17005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02c7358a95cb4757bedee7c5342322af: Bootstrap starting.
I20260925 15:13:20.617463 17008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:20.610236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39870 (local address 127.12.8.62:43719)
0925 15:13:20.611337 (+ 1101us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:20.611355 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:20.611402 (+ 47us) server_negotiation.cc:407] Connection header received
0925 15:13:20.613627 (+ 2225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:20.613652 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:20.614140 (+ 488us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:20.614568 (+ 428us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:20.615571 (+ 1003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:20.615579 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:20.615602 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:13:20.615622 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:20.615701 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:13:20.615725 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:20.615935 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:20.616058 (+ 123us) server_negotiation.cc:299] Negotiation successful
0925 15:13:20.616329 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":338,"threads_started":1}
I20260925 15:13:20.617756 17005 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 02c7358a95cb4757bedee7c5342322af: Neither blocks nor log segments found. Creating new log.
I20260925 15:13:20.618860 17005 log.cc:824] T 00000000000000000000000000000000 P 02c7358a95cb4757bedee7c5342322af: Log is configured to *not* fsync() on all Append() calls
I20260925 15:13:20.621619 17005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02c7358a95cb4757bedee7c5342322af: No bootstrap required, opened a new log
I20260925 15:13:20.628675 17005 raft_consensus.cc:348] T 00000000000000000000000000000000 P 02c7358a95cb4757bedee7c5342322af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02c7358a95cb4757bedee7c5342322af" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 43719 } }
I20260925 15:13:20.629016 17005 raft_consensus.cc:374] T 00000000000000000000000000000000 P 02c7358a95cb4757bedee7c5342322af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:13:20.629154 17005 raft_consensus.cc:729] T 00000000000000000000000000000000 P 02c7358a95cb4757bedee7c5342322af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02c7358a95cb4757bedee7c5342322af, State: Initialized, Role: FOLLOWER
I20260925 15:13:20.629742 17005 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02c7358a95cb4757bedee7c5342322af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02c7358a95cb4757bedee7c5342322af" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 43719 } }
I20260925 15:13:20.629976 17005 raft_consensus.cc:388] T 00000000000000000000000000000000 P 02c7358a95cb4757bedee7c5342322af [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:13:20.630107 17005 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02c7358a95cb4757bedee7c5342322af [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:13:20.630311 17005 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 02c7358a95cb4757bedee7c5342322af [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:13:20.632301 17005 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02c7358a95cb4757bedee7c5342322af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02c7358a95cb4757bedee7c5342322af" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 43719 } }
I20260925 15:13:20.632797 17005 leader_election.cc:302] T 00000000000000000000000000000000 P 02c7358a95cb4757bedee7c5342322af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 02c7358a95cb4757bedee7c5342322af; no voters:
I20260925 15:13:20.633406 17005 leader_election.cc:288] T 00000000000000000000000000000000 P 02c7358a95cb4757bedee7c5342322af [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:13:20.633528 17010 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 02c7358a95cb4757bedee7c5342322af [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:13:20.634254 17010 raft_consensus.cc:686] T 00000000000000000000000000000000 P 02c7358a95cb4757bedee7c5342322af [term 1 LEADER]: Becoming Leader. State: Replica: 02c7358a95cb4757bedee7c5342322af, State: Running, Role: LEADER
I20260925 15:13:20.634794 17010 consensus_queue.cc:237] T 00000000000000000000000000000000 P 02c7358a95cb4757bedee7c5342322af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02c7358a95cb4757bedee7c5342322af" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 43719 } }
I20260925 15:13:20.635483 17005 sys_catalog.cc:564] T 00000000000000000000000000000000 P 02c7358a95cb4757bedee7c5342322af [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:13:20.637346 17011 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02c7358a95cb4757bedee7c5342322af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "02c7358a95cb4757bedee7c5342322af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02c7358a95cb4757bedee7c5342322af" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 43719 } } }
I20260925 15:13:20.637698 17011 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02c7358a95cb4757bedee7c5342322af [sys.catalog]: This master's current role is: LEADER
I20260925 15:13:20.640987 17012 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02c7358a95cb4757bedee7c5342322af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02c7358a95cb4757bedee7c5342322af. Latest consensus state: current_term: 1 leader_uuid: "02c7358a95cb4757bedee7c5342322af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02c7358a95cb4757bedee7c5342322af" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 43719 } } }
I20260925 15:13:20.641264 17012 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02c7358a95cb4757bedee7c5342322af [sys.catalog]: This master's current role is: LEADER
I20260925 15:13:20.642354 17017 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:13:20.646826 17017 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:13:20.654505 17017 catalog_manager.cc:1347] Generated new cluster ID: 212a07e8684f432fa57017cb667faaf6
I20260925 15:13:20.654613 17017 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:13:20.671226 17017 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:13:20.671671 17017 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:13:20.677888 17017 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 02c7358a95cb4757bedee7c5342322af: Generated new TSK 0
I20260925 15:13:20.678689 17017 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:13:20.694083 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349162370577-12320-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349162370577-12320-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349162370577-12320-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349162370577-12320-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.1:0
--local_ip_for_outbound_sockets=127.12.8.1
--webserver_interface=127.12.8.1
--webserver_port=0
--tserver_master_addrs=127.12.8.62:43719
--builtin_ntp_servers=127.12.8.20:37867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 15:13:21.013592 17029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:13:21.013911 17029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:13:21.014063 17029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:13:21.021328 17029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:13:21.021477 17029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:13:21.021660 17029 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.1
I20260925 15:13:21.030616 17029 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:37867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349162370577-12320-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349162370577-12320-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349162370577-12320-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.8.1
--webserver_port=0
--tserver_master_addrs=127.12.8.62:43719
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.1
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349162370577-12320-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:13:21.031903 17029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:13:21.033627 17029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:13:21.047207 17034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:21.047101 17037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:21.047130 17035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:13:21.048012 17029 server_base.cc:1034] running on GCE node
I20260925 15:13:21.048566 17029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:13:21.049515 17029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:13:21.050736 17029 hybrid_clock.cc:648] HybridClock initialized: now 1790349201050678 us; error 53 us; skew 500 ppm
I20260925 15:13:21.051043 17029 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:13:21.053243 17029 webserver.cc:460] Webserver started at http://127.12.8.1:44235/ using document root <none> and password file <none>
I20260925 15:13:21.053905 17029 fs_manager.cc:359] Metadata directory not provided
I20260925 15:13:21.054037 17029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:13:21.054422 17029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:13:21.057111 17029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349162370577-12320-0/minicluster-data/ts-0/data/instance:
uuid: "4f2a34e3a4654bce9149a2a3b9506162"
format_stamp: "Formatted at 2026-09-25 15:13:21 on dist-test-slave-p468"
server_key: "fbec8110ff38a041809a574b0171c481"
server_key_iv: "c9f20649e4652adb6fbc31f12264c12d"
server_key_version: "encryptionkey@0"
I20260925 15:13:21.057899 17029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349162370577-12320-0/minicluster-data/ts-0/wal/instance:
uuid: "4f2a34e3a4654bce9149a2a3b9506162"
format_stamp: "Formatted at 2026-09-25 15:13:21 on dist-test-slave-p468"
server_key: "fbec8110ff38a041809a574b0171c481"
server_key_iv: "c9f20649e4652adb6fbc31f12264c12d"
server_key_version: "encryptionkey@0"
I20260925 15:13:21.062793 17029 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 15:13:21.065963 17043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:21.067212 17029 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:13:21.067412 17029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349162370577-12320-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349162370577-12320-0/minicluster-data/ts-0/wal
uuid: "4f2a34e3a4654bce9149a2a3b9506162"
format_stamp: "Formatted at 2026-09-25 15:13:21 on dist-test-slave-p468"
server_key: "fbec8110ff38a041809a574b0171c481"
server_key_iv: "c9f20649e4652adb6fbc31f12264c12d"
server_key_version: "encryptionkey@0"
I20260925 15:13:21.067634 17029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349162370577-12320-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349162370577-12320-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349162370577-12320-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:13:21.073174 17029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:13:21.073892 17029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:13:21.074146 17029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:13:21.075062 17029 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:13:21.076499 17029 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:13:21.076624 17029 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:21.076742 17029 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:13:21.076830 17029 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:21.108995 17029 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.1:39793
I20260925 15:13:21.109048 17155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.1:39793 every 8 connection(s)
I20260925 15:13:21.110322 17029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349162370577-12320-0/minicluster-data/ts-0/data/info.pb
I20260925 15:13:21.113791 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 17029
I20260925 15:13:21.113992 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349162370577-12320-0/minicluster-data/ts-0/wal/instance
I20260925 15:13:21.114871 12320 external_mini_cluster.cc:1428] Setting key d1c6ab3ad5128a6baab07d612b5beeab
I20260925 15:13:21.118222 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349162370577-12320-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349162370577-12320-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349162370577-12320-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349162370577-12320-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.2:0
--local_ip_for_outbound_sockets=127.12.8.2
--webserver_interface=127.12.8.2
--webserver_port=0
--tserver_master_addrs=127.12.8.62:43719
--builtin_ntp_servers=127.12.8.20:37867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260925 15:13:21.120908 17008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:21.113003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.1:41535 (local address 127.12.8.62:43719)
0925 15:13:21.113222 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:21.113230 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:21.114902 (+ 1672us) server_negotiation.cc:407] Connection header received
0925 15:13:21.118238 (+ 3336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:21.118250 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:21.118364 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:21.118488 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:21.120036 (+ 1548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:21.120045 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:21.120053 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:13:21.120077 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:21.120121 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:13:21.120139 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:21.120279 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:21.120503 (+ 224us) server_negotiation.cc:299] Negotiation successful
0925 15:13:21.120569 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 15:13:21.121644 17158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:21.113684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:43719 (local address 127.12.8.1:41535)
0925 15:13:21.114733 (+ 1049us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:21.114777 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:21.117033 (+ 2256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:21.118865 (+ 1832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:21.118874 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:21.118978 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:13:21.119728 (+ 750us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:13:21.119731 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:13:21.119770 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:13:21.119862 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:13:21.120258 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:13:21.120271 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:13:21.120278 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 15:13:21.120540 (+ 262us) client_negotiation.cc:240] Negotiation successful
0925 15:13:21.120756 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":397,"threads_started":1}
I20260925 15:13:21.123142 17156 heartbeater.cc:347] Connected to a master server at 127.12.8.62:43719
I20260925 15:13:21.123463 17156 heartbeater.cc:464] Registering TS with master...
I20260925 15:13:21.124049 17156 heartbeater.cc:511] Master 127.12.8.62:43719 requested a full tablet report, sending...
I20260925 15:13:21.125988 16970 ts_manager.cc:194] Registered new tserver with Master: 4f2a34e3a4654bce9149a2a3b9506162 (127.12.8.1:39793)
W20260925 15:13:21.436949 17159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:13:21.437196 17159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:13:21.437348 17159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:13:21.444478 17159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:13:21.444564 17159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:13:21.444731 17159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.2
I20260925 15:13:21.453848 17159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:37867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349162370577-12320-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349162370577-12320-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349162370577-12320-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.8.2
--webserver_port=0
--tserver_master_addrs=127.12.8.62:43719
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.2
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349162370577-12320-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:13:21.455149 17159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:13:21.456897 17159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:13:21.469655 17165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:21.469625 17164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:21.469611 17167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:13:21.470650 17159 server_base.cc:1034] running on GCE node
I20260925 15:13:21.471267 17159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:13:21.472334 17159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:13:21.473532 17159 hybrid_clock.cc:648] HybridClock initialized: now 1790349201473473 us; error 55 us; skew 500 ppm
I20260925 15:13:21.473843 17159 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:13:21.476181 17159 webserver.cc:460] Webserver started at http://127.12.8.2:38621/ using document root <none> and password file <none>
I20260925 15:13:21.476868 17159 fs_manager.cc:359] Metadata directory not provided
I20260925 15:13:21.476984 17159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:13:21.477321 17159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:13:21.480140 17159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349162370577-12320-0/minicluster-data/ts-1/data/instance:
uuid: "591bfab4e4054c12bb6d35f58786e750"
format_stamp: "Formatted at 2026-09-25 15:13:21 on dist-test-slave-p468"
server_key: "338b17addf2a800d30116cb8485c5463"
server_key_iv: "d8192e69f8843446ead35221fba27fbc"
server_key_version: "encryptionkey@0"
I20260925 15:13:21.480914 17159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349162370577-12320-0/minicluster-data/ts-1/wal/instance:
uuid: "591bfab4e4054c12bb6d35f58786e750"
format_stamp: "Formatted at 2026-09-25 15:13:21 on dist-test-slave-p468"
server_key: "338b17addf2a800d30116cb8485c5463"
server_key_iv: "d8192e69f8843446ead35221fba27fbc"
server_key_version: "encryptionkey@0"
I20260925 15:13:21.485992 17159 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260925 15:13:21.489319 17173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:21.490789 17159 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:13:21.490960 17159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349162370577-12320-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349162370577-12320-0/minicluster-data/ts-1/wal
uuid: "591bfab4e4054c12bb6d35f58786e750"
format_stamp: "Formatted at 2026-09-25 15:13:21 on dist-test-slave-p468"
server_key: "338b17addf2a800d30116cb8485c5463"
server_key_iv: "d8192e69f8843446ead35221fba27fbc"
server_key_version: "encryptionkey@0"
I20260925 15:13:21.491163 17159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349162370577-12320-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349162370577-12320-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349162370577-12320-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:13:21.496764 17159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:13:21.497520 17159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:13:21.497787 17159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:13:21.498746 17159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:13:21.500062 17159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:13:21.500144 17159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:21.500305 17159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:13:21.500396 17159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:21.530228 17159 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.2:37611
I20260925 15:13:21.530316 17285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.2:37611 every 8 connection(s)
I20260925 15:13:21.531512 17159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349162370577-12320-0/minicluster-data/ts-1/data/info.pb
I20260925 15:13:21.537297 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 17159
I20260925 15:13:21.537457 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349162370577-12320-0/minicluster-data/ts-1/wal/instance
I20260925 15:13:21.537909 12320 external_mini_cluster.cc:1428] Setting key 19a13d87f500aa271a3b469262767e49
I20260925 15:13:21.540164 17008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:21.534011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.2:45813 (local address 127.12.8.62:43719)
0925 15:13:21.534242 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:21.534248 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:21.535792 (+ 1544us) server_negotiation.cc:407] Connection header received
0925 15:13:21.537687 (+ 1895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:21.537692 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:21.537767 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:21.537878 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:21.539376 (+ 1498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:21.539381 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:21.539388 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:13:21.539409 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:21.539439 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:13:21.539450 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:21.539603 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:21.539791 (+ 188us) server_negotiation.cc:299] Negotiation successful
0925 15:13:21.539856 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 15:13:21.540750 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349162370577-12320-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349162370577-12320-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349162370577-12320-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349162370577-12320-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.3:0
--local_ip_for_outbound_sockets=127.12.8.3
--webserver_interface=127.12.8.3
--webserver_port=0
--tserver_master_addrs=127.12.8.62:43719
--builtin_ntp_servers=127.12.8.20:37867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260925 15:13:21.541124 17288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:21.534621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:43719 (local address 127.12.8.2:45813)
0925 15:13:21.535637 (+ 1016us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:21.535673 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:21.537343 (+ 1670us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:21.538116 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:21.538125 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:21.538256 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:13:21.538999 (+ 743us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:13:21.539001 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:13:21.539037 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:13:21.539149 (+ 112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:13:21.539549 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:13:21.539561 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:13:21.539570 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 15:13:21.539864 (+ 294us) client_negotiation.cc:240] Negotiation successful
0925 15:13:21.540110 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":269,"threads_started":1}
I20260925 15:13:21.542594 17286 heartbeater.cc:347] Connected to a master server at 127.12.8.62:43719
I20260925 15:13:21.542897 17286 heartbeater.cc:464] Registering TS with master...
I20260925 15:13:21.543429 17286 heartbeater.cc:511] Master 127.12.8.62:43719 requested a full tablet report, sending...
I20260925 15:13:21.544729 16970 ts_manager.cc:194] Registered new tserver with Master: 591bfab4e4054c12bb6d35f58786e750 (127.12.8.2:37611)
W20260925 15:13:21.854907 17289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:13:21.855196 17289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:13:21.855356 17289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:13:21.866314 17289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:13:21.866441 17289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:13:21.866622 17289 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.3
I20260925 15:13:21.876080 17289 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:37867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349162370577-12320-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349162370577-12320-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349162370577-12320-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.8.3
--webserver_port=0
--tserver_master_addrs=127.12.8.62:43719
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.3
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349162370577-12320-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:13:21.877336 17289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:13:21.879096 17289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:13:21.892164 17294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:21.892216 17297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:21.892164 17295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:13:21.893070 17289 server_base.cc:1034] running on GCE node
I20260925 15:13:21.893682 17289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:13:21.894742 17289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:13:21.895960 17289 hybrid_clock.cc:648] HybridClock initialized: now 1790349201895903 us; error 49 us; skew 500 ppm
I20260925 15:13:21.896277 17289 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:13:21.898567 17289 webserver.cc:460] Webserver started at http://127.12.8.3:36595/ using document root <none> and password file <none>
I20260925 15:13:21.899230 17289 fs_manager.cc:359] Metadata directory not provided
I20260925 15:13:21.899369 17289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:13:21.899684 17289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:13:21.902442 17289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349162370577-12320-0/minicluster-data/ts-2/data/instance:
uuid: "fb03b8b8b56e41fb8d81c6d96d959772"
format_stamp: "Formatted at 2026-09-25 15:13:21 on dist-test-slave-p468"
server_key: "3dc19b2c18cc2c608f78b88e507bd801"
server_key_iv: "502cbb7a1a828c2db72d86363ac58bef"
server_key_version: "encryptionkey@0"
I20260925 15:13:21.903216 17289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349162370577-12320-0/minicluster-data/ts-2/wal/instance:
uuid: "fb03b8b8b56e41fb8d81c6d96d959772"
format_stamp: "Formatted at 2026-09-25 15:13:21 on dist-test-slave-p468"
server_key: "3dc19b2c18cc2c608f78b88e507bd801"
server_key_iv: "502cbb7a1a828c2db72d86363ac58bef"
server_key_version: "encryptionkey@0"
I20260925 15:13:21.908077 17289 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 15:13:21.911381 17303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:21.912608 17289 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:13:21.912768 17289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349162370577-12320-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349162370577-12320-0/minicluster-data/ts-2/wal
uuid: "fb03b8b8b56e41fb8d81c6d96d959772"
format_stamp: "Formatted at 2026-09-25 15:13:21 on dist-test-slave-p468"
server_key: "3dc19b2c18cc2c608f78b88e507bd801"
server_key_iv: "502cbb7a1a828c2db72d86363ac58bef"
server_key_version: "encryptionkey@0"
I20260925 15:13:21.912971 17289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349162370577-12320-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349162370577-12320-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349162370577-12320-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:13:21.918373 17289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:13:21.919101 17289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:13:21.919368 17289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:13:21.920236 17289 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:13:21.921578 17289 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:13:21.921697 17289 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:21.921829 17289 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:13:21.921916 17289 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:21.954779 17289 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.3:34067
I20260925 15:13:21.954829 17415 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.3:34067 every 8 connection(s)
I20260925 15:13:21.956079 17289 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349162370577-12320-0/minicluster-data/ts-2/data/info.pb
I20260925 15:13:21.958613 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 17289
I20260925 15:13:21.958781 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349162370577-12320-0/minicluster-data/ts-2/wal/instance
I20260925 15:13:21.959165 12320 external_mini_cluster.cc:1428] Setting key 17ebb10632e6064aa55292a47a51f22b
I20260925 15:13:21.964131 17008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:21.958627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.3:42135 (local address 127.12.8.62:43719)
0925 15:13:21.958871 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:21.958877 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:21.960336 (+ 1459us) server_negotiation.cc:407] Connection header received
0925 15:13:21.962353 (+ 2017us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:21.962357 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:21.962408 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:21.962488 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:21.963593 (+ 1105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:21.963596 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:21.963599 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:13:21.963613 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:21.963642 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:13:21.963653 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:21.963738 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:21.963898 (+ 160us) server_negotiation.cc:299] Negotiation successful
0925 15:13:21.963943 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 15:13:21.965003 17418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:21.959166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:43719 (local address 127.12.8.3:42135)
0925 15:13:21.960163 (+ 997us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:21.960217 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:21.962063 (+ 1846us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:21.962674 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:21.962682 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:21.962767 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:13:21.963340 (+ 573us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:13:21.963342 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:13:21.963378 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:13:21.963455 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:13:21.963734 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:13:21.963741 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:13:21.963746 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 15:13:21.963943 (+ 197us) client_negotiation.cc:240] Negotiation successful
0925 15:13:21.964118 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":308,"threads_started":1}
I20260925 15:13:21.966348 17416 heartbeater.cc:347] Connected to a master server at 127.12.8.62:43719
I20260925 15:13:21.966656 17416 heartbeater.cc:464] Registering TS with master...
I20260925 15:13:21.967222 17416 heartbeater.cc:511] Master 127.12.8.62:43719 requested a full tablet report, sending...
I20260925 15:13:21.968537 16970 ts_manager.cc:194] Registered new tserver with Master: fb03b8b8b56e41fb8d81c6d96d959772 (127.12.8.3:34067)
I20260925 15:13:21.974258 12320 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:13:21.980921 17008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:21.978662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39886 (local address 127.12.8.62:43719)
0925 15:13:21.978859 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:21.978868 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:21.979362 (+ 494us) server_negotiation.cc:407] Connection header received
0925 15:13:21.979479 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:21.979482 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:21.979528 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:21.979618 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:21.980402 (+ 784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:21.980406 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:21.980409 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:13:21.980423 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:21.980451 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:13:21.980461 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:21.980558 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:21.980591 (+ 33us) server_negotiation.cc:299] Negotiation successful
0925 15:13:21.980630 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 15:13:21.984550 16970 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39886:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:13:21.991110 16970 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:13:22.008077 17428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:22.002634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.3:34067 (local address 127.0.0.1:57104)
0925 15:13:22.003364 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:22.003415 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:22.003533 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:22.004661 (+ 1128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:22.004672 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:22.004794 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:13:22.005803 (+ 1009us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:13:22.005809 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:13:22.005846 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:13:22.005945 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:13:22.006728 (+ 783us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:13:22.006743 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:13:22.006755 (+ 12us) client_negotiation.cc:769] Sending connection context
0925 15:13:22.006960 (+ 205us) client_negotiation.cc:240] Negotiation successful
0925 15:13:22.007063 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":303,"threads_started":1}
I20260925 15:13:22.009159 17429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:22.002915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57104 (local address 127.12.8.3:34067)
0925 15:13:22.003593 (+ 678us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:22.003603 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:22.003638 (+ 35us) server_negotiation.cc:407] Connection header received
0925 15:13:22.003710 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:22.003718 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:22.003972 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:22.004129 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:22.006416 (+ 2287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:22.006429 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:22.006448 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:13:22.006474 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:22.006548 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:13:22.006574 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:22.007434 (+ 860us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:22.007513 (+ 79us) server_negotiation.cc:299] Negotiation successful
0925 15:13:22.007629 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":714,"threads_started":1}
I20260925 15:13:22.009985 17431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:22.003478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.1:39793 (local address 127.0.0.1:35522)
0925 15:13:22.004330 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:22.004351 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:22.004461 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:22.007342 (+ 2881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:22.007347 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:22.007368 (+ 21us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:13:22.008494 (+ 1126us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:13:22.008497 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:13:22.008513 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:13:22.008559 (+ 46us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:13:22.009413 (+ 854us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:13:22.009419 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:13:22.009424 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 15:13:22.009571 (+ 147us) client_negotiation.cc:240] Negotiation successful
0925 15:13:22.009617 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"thread_start_us":3763,"threads_started":1}
I20260925 15:13:22.012212 17432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:22.004217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35522 (local address 127.12.8.1:39793)
0925 15:13:22.006551 (+ 2334us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:22.006564 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:22.006597 (+ 33us) server_negotiation.cc:407] Connection header received
0925 15:13:22.006674 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:22.006685 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:22.006940 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:22.007460 (+ 520us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:22.008730 (+ 1270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:22.008742 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:22.008770 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:13:22.008785 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:22.008830 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:13:22.008848 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:22.008989 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:22.009583 (+ 594us) server_negotiation.cc:299] Negotiation successful
0925 15:13:22.009704 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2140,"thread_start_us":2029,"threads_started":1}
I20260925 15:13:22.013661 17351 tablet_service.cc:1467] Processing CreateTablet for tablet 7f616d23efed46418c8f1a9455ce5f3e (DEFAULT_TABLE table=test-table [id=ab20debf03754f459a9b731a00a48676]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:13:22.015079 17351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f616d23efed46418c8f1a9455ce5f3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:13:22.015990 17430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:22.003742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.2:37611 (local address 127.0.0.1:43630)
0925 15:13:22.008494 (+ 4752us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:22.008510 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:22.008593 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:22.013409 (+ 4816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:22.013414 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:22.013438 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:13:22.014099 (+ 661us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:13:22.014103 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:13:22.014118 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:13:22.014164 (+ 46us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:13:22.015400 (+ 1236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:13:22.015407 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:13:22.015415 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 15:13:22.015573 (+ 158us) client_negotiation.cc:240] Negotiation successful
0925 15:13:22.015626 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4627,"thread_start_us":1529,"threads_started":1}
I20260925 15:13:22.016057 17433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:22.004608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43630 (local address 127.12.8.2:37611)
0925 15:13:22.012871 (+ 8263us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:22.012879 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:22.012907 (+ 28us) server_negotiation.cc:407] Connection header received
0925 15:13:22.012976 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:22.012983 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:22.013200 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:22.013832 (+ 632us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:22.015180 (+ 1348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:22.015188 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:22.015200 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:13:22.015215 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:22.015265 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:13:22.015286 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:22.015448 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:22.015572 (+ 124us) server_negotiation.cc:299] Negotiation successful
0925 15:13:22.015710 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8089,"thread_start_us":7965,"threads_started":1}
I20260925 15:13:22.016572 17091 tablet_service.cc:1467] Processing CreateTablet for tablet 7f616d23efed46418c8f1a9455ce5f3e (DEFAULT_TABLE table=test-table [id=ab20debf03754f459a9b731a00a48676]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:13:22.017797 17091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f616d23efed46418c8f1a9455ce5f3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:13:22.020354 17221 tablet_service.cc:1467] Processing CreateTablet for tablet 7f616d23efed46418c8f1a9455ce5f3e (DEFAULT_TABLE table=test-table [id=ab20debf03754f459a9b731a00a48676]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:13:22.021566 17221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f616d23efed46418c8f1a9455ce5f3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:13:22.026320 17434 tablet_bootstrap.cc:492] T 7f616d23efed46418c8f1a9455ce5f3e P fb03b8b8b56e41fb8d81c6d96d959772: Bootstrap starting.
I20260925 15:13:22.029724 17434 tablet_bootstrap.cc:654] T 7f616d23efed46418c8f1a9455ce5f3e P fb03b8b8b56e41fb8d81c6d96d959772: Neither blocks nor log segments found. Creating new log.
I20260925 15:13:22.030761 17434 log.cc:824] T 7f616d23efed46418c8f1a9455ce5f3e P fb03b8b8b56e41fb8d81c6d96d959772: Log is configured to *not* fsync() on all Append() calls
I20260925 15:13:22.031685 17435 tablet_bootstrap.cc:492] T 7f616d23efed46418c8f1a9455ce5f3e P 4f2a34e3a4654bce9149a2a3b9506162: Bootstrap starting.
I20260925 15:13:22.034803 17437 tablet_bootstrap.cc:492] T 7f616d23efed46418c8f1a9455ce5f3e P 591bfab4e4054c12bb6d35f58786e750: Bootstrap starting.
I20260925 15:13:22.034880 17434 tablet_bootstrap.cc:492] T 7f616d23efed46418c8f1a9455ce5f3e P fb03b8b8b56e41fb8d81c6d96d959772: No bootstrap required, opened a new log
I20260925 15:13:22.035465 17434 ts_tablet_manager.cc:1397] T 7f616d23efed46418c8f1a9455ce5f3e P fb03b8b8b56e41fb8d81c6d96d959772: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260925 15:13:22.036482 17435 tablet_bootstrap.cc:654] T 7f616d23efed46418c8f1a9455ce5f3e P 4f2a34e3a4654bce9149a2a3b9506162: Neither blocks nor log segments found. Creating new log.
I20260925 15:13:22.037564 17437 tablet_bootstrap.cc:654] T 7f616d23efed46418c8f1a9455ce5f3e P 591bfab4e4054c12bb6d35f58786e750: Neither blocks nor log segments found. Creating new log.
I20260925 15:13:22.037679 17435 log.cc:824] T 7f616d23efed46418c8f1a9455ce5f3e P 4f2a34e3a4654bce9149a2a3b9506162: Log is configured to *not* fsync() on all Append() calls
I20260925 15:13:22.038761 17437 log.cc:824] T 7f616d23efed46418c8f1a9455ce5f3e P 591bfab4e4054c12bb6d35f58786e750: Log is configured to *not* fsync() on all Append() calls
I20260925 15:13:22.040910 17435 tablet_bootstrap.cc:492] T 7f616d23efed46418c8f1a9455ce5f3e P 4f2a34e3a4654bce9149a2a3b9506162: No bootstrap required, opened a new log
I20260925 15:13:22.041231 17435 ts_tablet_manager.cc:1397] T 7f616d23efed46418c8f1a9455ce5f3e P 4f2a34e3a4654bce9149a2a3b9506162: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260925 15:13:22.044044 17434 raft_consensus.cc:348] T 7f616d23efed46418c8f1a9455ce5f3e P fb03b8b8b56e41fb8d81c6d96d959772 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb03b8b8b56e41fb8d81c6d96d959772" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 34067 } } peers { permanent_uuid: "591bfab4e4054c12bb6d35f58786e750" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 37611 } } peers { permanent_uuid: "4f2a34e3a4654bce9149a2a3b9506162" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 39793 } }
I20260925 15:13:22.044443 17434 raft_consensus.cc:374] T 7f616d23efed46418c8f1a9455ce5f3e P fb03b8b8b56e41fb8d81c6d96d959772 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:13:22.044570 17434 raft_consensus.cc:729] T 7f616d23efed46418c8f1a9455ce5f3e P fb03b8b8b56e41fb8d81c6d96d959772 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb03b8b8b56e41fb8d81c6d96d959772, State: Initialized, Role: FOLLOWER
I20260925 15:13:22.045149 17434 consensus_queue.cc:260] T 7f616d23efed46418c8f1a9455ce5f3e P fb03b8b8b56e41fb8d81c6d96d959772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb03b8b8b56e41fb8d81c6d96d959772" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 34067 } } peers { permanent_uuid: "591bfab4e4054c12bb6d35f58786e750" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 37611 } } peers { permanent_uuid: "4f2a34e3a4654bce9149a2a3b9506162" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 39793 } }
I20260925 15:13:22.046168 17416 heartbeater.cc:503] Master 127.12.8.62:43719 was elected leader, sending a full tablet report...
I20260925 15:13:22.047329 17434 ts_tablet_manager.cc:1428] T 7f616d23efed46418c8f1a9455ce5f3e P fb03b8b8b56e41fb8d81c6d96d959772: Time spent starting tablet: real 0.012s user 0.010s sys 0.000s
I20260925 15:13:22.047951 17437 tablet_bootstrap.cc:492] T 7f616d23efed46418c8f1a9455ce5f3e P 591bfab4e4054c12bb6d35f58786e750: No bootstrap required, opened a new log
I20260925 15:13:22.048210 17437 ts_tablet_manager.cc:1397] T 7f616d23efed46418c8f1a9455ce5f3e P 591bfab4e4054c12bb6d35f58786e750: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.004s
I20260925 15:13:22.051453 17435 raft_consensus.cc:348] T 7f616d23efed46418c8f1a9455ce5f3e P 4f2a34e3a4654bce9149a2a3b9506162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb03b8b8b56e41fb8d81c6d96d959772" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 34067 } } peers { permanent_uuid: "591bfab4e4054c12bb6d35f58786e750" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 37611 } } peers { permanent_uuid: "4f2a34e3a4654bce9149a2a3b9506162" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 39793 } }
I20260925 15:13:22.051797 17435 raft_consensus.cc:374] T 7f616d23efed46418c8f1a9455ce5f3e P 4f2a34e3a4654bce9149a2a3b9506162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:13:22.051913 17435 raft_consensus.cc:729] T 7f616d23efed46418c8f1a9455ce5f3e P 4f2a34e3a4654bce9149a2a3b9506162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f2a34e3a4654bce9149a2a3b9506162, State: Initialized, Role: FOLLOWER
I20260925 15:13:22.052418 17440 raft_consensus.cc:482] T 7f616d23efed46418c8f1a9455ce5f3e P fb03b8b8b56e41fb8d81c6d96d959772 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:13:22.052613 17435 consensus_queue.cc:260] T 7f616d23efed46418c8f1a9455ce5f3e P 4f2a34e3a4654bce9149a2a3b9506162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb03b8b8b56e41fb8d81c6d96d959772" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 34067 } } peers { permanent_uuid: "591bfab4e4054c12bb6d35f58786e750" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 37611 } } peers { permanent_uuid: "4f2a34e3a4654bce9149a2a3b9506162" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 39793 } }
I20260925 15:13:22.052654 17440 raft_consensus.cc:504] T 7f616d23efed46418c8f1a9455ce5f3e P fb03b8b8b56e41fb8d81c6d96d959772 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb03b8b8b56e41fb8d81c6d96d959772" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 34067 } } peers { permanent_uuid: "591bfab4e4054c12bb6d35f58786e750" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 37611 } } peers { permanent_uuid: "4f2a34e3a4654bce9149a2a3b9506162" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 39793 } }
I20260925 15:13:22.053638 17156 heartbeater.cc:503] Master 127.12.8.62:43719 was elected leader, sending a full tablet report...
I20260925 15:13:22.054324 17435 ts_tablet_manager.cc:1428] T 7f616d23efed46418c8f1a9455ce5f3e P 4f2a34e3a4654bce9149a2a3b9506162: Time spent starting tablet: real 0.013s user 0.014s sys 0.001s
I20260925 15:13:22.057184 17418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:22.054666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.2:37611 (local address 127.12.8.3:55619)
0925 15:13:22.054863 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:22.054884 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:22.054965 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:22.055665 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:22.055670 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:22.055699 (+ 29us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:13:22.056325 (+ 626us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:13:22.056327 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:13:22.056341 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:13:22.056376 (+ 35us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:13:22.056796 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:13:22.056800 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:13:22.056805 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 15:13:22.056902 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 15:13:22.056951 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260925 15:13:22.058409 17433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:22.055065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.3:55619 (local address 127.12.8.2:37611)
0925 15:13:22.055250 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:22.055256 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:22.055275 (+ 19us) server_negotiation.cc:407] Connection header received
0925 15:13:22.055330 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:22.055335 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:22.055387 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:22.055490 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:22.056543 (+ 1053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:22.056549 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:22.056553 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:13:22.056572 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:22.056609 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:13:22.056624 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:22.056735 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:22.058061 (+ 1326us) server_negotiation.cc:299] Negotiation successful
0925 15:13:22.058115 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 15:13:22.059355 17241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f616d23efed46418c8f1a9455ce5f3e" candidate_uuid: "fb03b8b8b56e41fb8d81c6d96d959772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "591bfab4e4054c12bb6d35f58786e750" is_pre_election: true
I20260925 15:13:22.059950 17437 raft_consensus.cc:348] T 7f616d23efed46418c8f1a9455ce5f3e P 591bfab4e4054c12bb6d35f58786e750 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb03b8b8b56e41fb8d81c6d96d959772" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 34067 } } peers { permanent_uuid: "591bfab4e4054c12bb6d35f58786e750" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 37611 } } peers { permanent_uuid: "4f2a34e3a4654bce9149a2a3b9506162" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 39793 } }
I20260925 15:13:22.060428 17437 raft_consensus.cc:374] T 7f616d23efed46418c8f1a9455ce5f3e P 591bfab4e4054c12bb6d35f58786e750 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:13:22.060528 17437 raft_consensus.cc:729] T 7f616d23efed46418c8f1a9455ce5f3e P 591bfab4e4054c12bb6d35f58786e750 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 591bfab4e4054c12bb6d35f58786e750, State: Initialized, Role: FOLLOWER
W20260925 15:13:22.060770 17304 leader_election.cc:341] T 7f616d23efed46418c8f1a9455ce5f3e P fb03b8b8b56e41fb8d81c6d96d959772 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 591bfab4e4054c12bb6d35f58786e750 (127.12.8.2:37611): Illegal state: must be running to vote when last-logged opid is not known
I20260925 15:13:22.061007 17418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:22.055626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.1:39793 (local address 127.12.8.3:35201)
0925 15:13:22.057411 (+ 1785us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:22.057427 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:22.057503 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:22.057811 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:22.057817 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:22.057838 (+ 21us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:13:22.058427 (+ 589us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:13:22.058432 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:13:22.058448 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:13:22.059876 (+ 1428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:13:22.060285 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:13:22.060290 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:13:22.060297 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 15:13:22.060400 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 15:13:22.060449 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1696,"threads_started":1}
I20260925 15:13:22.061214 17432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:22.056382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.3:35201 (local address 127.12.8.1:39793)
0925 15:13:22.056553 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:22.056559 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:22.057591 (+ 1032us) server_negotiation.cc:407] Connection header received
0925 15:13:22.057634 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:22.057637 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:22.057674 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:22.057781 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:22.060035 (+ 2254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:22.060041 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:22.060045 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:13:22.060063 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:22.060096 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:13:22.060111 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:22.060221 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:22.060864 (+ 643us) server_negotiation.cc:299] Negotiation successful
0925 15:13:22.060923 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 15:13:22.061329 17437 consensus_queue.cc:260] T 7f616d23efed46418c8f1a9455ce5f3e P 591bfab4e4054c12bb6d35f58786e750 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb03b8b8b56e41fb8d81c6d96d959772" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 34067 } } peers { permanent_uuid: "591bfab4e4054c12bb6d35f58786e750" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 37611 } } peers { permanent_uuid: "4f2a34e3a4654bce9149a2a3b9506162" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 39793 } }
I20260925 15:13:22.061550 17440 leader_election.cc:288] T 7f616d23efed46418c8f1a9455ce5f3e P fb03b8b8b56e41fb8d81c6d96d959772 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 591bfab4e4054c12bb6d35f58786e750 (127.12.8.2:37611), 4f2a34e3a4654bce9149a2a3b9506162 (127.12.8.1:39793)
I20260925 15:13:22.062485 17286 heartbeater.cc:503] Master 127.12.8.62:43719 was elected leader, sending a full tablet report...
I20260925 15:13:22.063270 17437 ts_tablet_manager.cc:1428] T 7f616d23efed46418c8f1a9455ce5f3e P 591bfab4e4054c12bb6d35f58786e750: Time spent starting tablet: real 0.015s user 0.010s sys 0.005s
I20260925 15:13:22.067551 17111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f616d23efed46418c8f1a9455ce5f3e" candidate_uuid: "fb03b8b8b56e41fb8d81c6d96d959772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f2a34e3a4654bce9149a2a3b9506162" is_pre_election: true
I20260925 15:13:22.067950 17111 raft_consensus.cc:2454] T 7f616d23efed46418c8f1a9455ce5f3e P 4f2a34e3a4654bce9149a2a3b9506162 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb03b8b8b56e41fb8d81c6d96d959772 in term 0.
I20260925 15:13:22.068425 17307 leader_election.cc:302] T 7f616d23efed46418c8f1a9455ce5f3e P fb03b8b8b56e41fb8d81c6d96d959772 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4f2a34e3a4654bce9149a2a3b9506162, fb03b8b8b56e41fb8d81c6d96d959772; no voters: 591bfab4e4054c12bb6d35f58786e750
I20260925 15:13:22.068773 17440 raft_consensus.cc:2781] T 7f616d23efed46418c8f1a9455ce5f3e P fb03b8b8b56e41fb8d81c6d96d959772 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:13:22.068913 17440 raft_consensus.cc:482] T 7f616d23efed46418c8f1a9455ce5f3e P fb03b8b8b56e41fb8d81c6d96d959772 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:13:22.069015 17440 raft_consensus.cc:3037] T 7f616d23efed46418c8f1a9455ce5f3e P fb03b8b8b56e41fb8d81c6d96d959772 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:13:22.071208 17440 raft_consensus.cc:504] T 7f616d23efed46418c8f1a9455ce5f3e P fb03b8b8b56e41fb8d81c6d96d959772 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb03b8b8b56e41fb8d81c6d96d959772" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 34067 } } peers { permanent_uuid: "591bfab4e4054c12bb6d35f58786e750" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 37611 } } peers { permanent_uuid: "4f2a34e3a4654bce9149a2a3b9506162" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 39793 } }
I20260925 15:13:22.071831 17440 leader_election.cc:288] T 7f616d23efed46418c8f1a9455ce5f3e P fb03b8b8b56e41fb8d81c6d96d959772 [CANDIDATE]: Term 1 election: Requested vote from peers 591bfab4e4054c12bb6d35f58786e750 (127.12.8.2:37611), 4f2a34e3a4654bce9149a2a3b9506162 (127.12.8.1:39793)
I20260925 15:13:22.072168 17241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f616d23efed46418c8f1a9455ce5f3e" candidate_uuid: "fb03b8b8b56e41fb8d81c6d96d959772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "591bfab4e4054c12bb6d35f58786e750"
I20260925 15:13:22.072297 17111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f616d23efed46418c8f1a9455ce5f3e" candidate_uuid: "fb03b8b8b56e41fb8d81c6d96d959772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f2a34e3a4654bce9149a2a3b9506162"
I20260925 15:13:22.072438 17241 raft_consensus.cc:3037] T 7f616d23efed46418c8f1a9455ce5f3e P 591bfab4e4054c12bb6d35f58786e750 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:13:22.072477 17111 raft_consensus.cc:3037] T 7f616d23efed46418c8f1a9455ce5f3e P 4f2a34e3a4654bce9149a2a3b9506162 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:13:22.075642 17241 raft_consensus.cc:2454] T 7f616d23efed46418c8f1a9455ce5f3e P 591bfab4e4054c12bb6d35f58786e750 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb03b8b8b56e41fb8d81c6d96d959772 in term 1.
I20260925 15:13:22.075649 17111 raft_consensus.cc:2454] T 7f616d23efed46418c8f1a9455ce5f3e P 4f2a34e3a4654bce9149a2a3b9506162 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb03b8b8b56e41fb8d81c6d96d959772 in term 1.
I20260925 15:13:22.076022 17304 leader_election.cc:302] T 7f616d23efed46418c8f1a9455ce5f3e P fb03b8b8b56e41fb8d81c6d96d959772 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 591bfab4e4054c12bb6d35f58786e750, fb03b8b8b56e41fb8d81c6d96d959772; no voters:
I20260925 15:13:22.076308 17440 raft_consensus.cc:2781] T 7f616d23efed46418c8f1a9455ce5f3e P fb03b8b8b56e41fb8d81c6d96d959772 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:13:22.076754 17440 raft_consensus.cc:686] T 7f616d23efed46418c8f1a9455ce5f3e P fb03b8b8b56e41fb8d81c6d96d959772 [term 1 LEADER]: Becoming Leader. State: Replica: fb03b8b8b56e41fb8d81c6d96d959772, State: Running, Role: LEADER
I20260925 15:13:22.077260 17440 consensus_queue.cc:237] T 7f616d23efed46418c8f1a9455ce5f3e P fb03b8b8b56e41fb8d81c6d96d959772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb03b8b8b56e41fb8d81c6d96d959772" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 34067 } } peers { permanent_uuid: "591bfab4e4054c12bb6d35f58786e750" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 37611 } } peers { permanent_uuid: "4f2a34e3a4654bce9149a2a3b9506162" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 39793 } }
I20260925 15:13:22.081228 16969 catalog_manager.cc:5495] T 7f616d23efed46418c8f1a9455ce5f3e P fb03b8b8b56e41fb8d81c6d96d959772 reported cstate change: term changed from 0 to 1, leader changed from <none> to fb03b8b8b56e41fb8d81c6d96d959772 (127.12.8.3). New cstate: current_term: 1 leader_uuid: "fb03b8b8b56e41fb8d81c6d96d959772" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb03b8b8b56e41fb8d81c6d96d959772" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 34067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "591bfab4e4054c12bb6d35f58786e750" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 37611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f2a34e3a4654bce9149a2a3b9506162" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 39793 } health_report { overall_health: UNKNOWN } } }
I20260925 15:13:22.112003 17429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:22.109411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57116 (local address 127.12.8.3:34067)
0925 15:13:22.109593 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:22.109601 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:22.109771 (+ 170us) server_negotiation.cc:407] Connection header received
0925 15:13:22.110026 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:22.110031 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:22.110133 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:22.110289 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:22.111398 (+ 1109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:22.111403 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:22.111407 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:13:22.111424 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:22.111457 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:13:22.111469 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:22.111610 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:22.111699 (+ 89us) server_negotiation.cc:299] Negotiation successful
0925 15:13:22.111761 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260925 15:13:22.112483 17157 tablet.cc:2369] T 7f616d23efed46418c8f1a9455ce5f3e P 4f2a34e3a4654bce9149a2a3b9506162: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:13:22.121670 17111 raft_consensus.cc:1261] T 7f616d23efed46418c8f1a9455ce5f3e P 4f2a34e3a4654bce9149a2a3b9506162 [term 1 FOLLOWER]: Refusing update from remote peer fb03b8b8b56e41fb8d81c6d96d959772: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:13:22.121636 17241 raft_consensus.cc:1261] T 7f616d23efed46418c8f1a9455ce5f3e P 591bfab4e4054c12bb6d35f58786e750 [term 1 FOLLOWER]: Refusing update from remote peer fb03b8b8b56e41fb8d81c6d96d959772: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:13:22.122571 17444 consensus_queue.cc:1035] T 7f616d23efed46418c8f1a9455ce5f3e P fb03b8b8b56e41fb8d81c6d96d959772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f2a34e3a4654bce9149a2a3b9506162" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 39793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:13:22.123026 17440 consensus_queue.cc:1035] T 7f616d23efed46418c8f1a9455ce5f3e P fb03b8b8b56e41fb8d81c6d96d959772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "591bfab4e4054c12bb6d35f58786e750" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 37611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:13:22.135540 17448 mvcc.cc:205] Tried to move back new op lower bound from 7333270331877494784 to 7333270331711922176. Current Snapshot: MvccSnapshot[applied={T|T < 7333270331877494784}]
I20260925 15:13:22.136274 17449 mvcc.cc:205] Tried to move back new op lower bound from 7333270331877494784 to 7333270331711922176. Current Snapshot: MvccSnapshot[applied={T|T < 7333270331877494784}]
I20260925 15:13:22.147810 16969 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39886:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:13:22.148090 16969 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39886:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:13:22.151578 16969 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 02c7358a95cb4757bedee7c5342322af: Sending DeleteTablet for 3 replicas of tablet 7f616d23efed46418c8f1a9455ce5f3e
I20260925 15:13:22.152568 17351 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f616d23efed46418c8f1a9455ce5f3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:13:22 UTC) from {username='slave'} at 127.0.0.1:57104
I20260925 15:13:22.152756 17221 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f616d23efed46418c8f1a9455ce5f3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:13:22 UTC) from {username='slave'} at 127.0.0.1:43630
I20260925 15:13:22.152808 17091 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f616d23efed46418c8f1a9455ce5f3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:13:22 UTC) from {username='slave'} at 127.0.0.1:35522
I20260925 15:13:22.154559 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 17029
I20260925 15:13:22.164963 17456 tablet_replica.cc:330] T 7f616d23efed46418c8f1a9455ce5f3e P fb03b8b8b56e41fb8d81c6d96d959772: stopping tablet replica
I20260925 15:13:22.165616 17456 raft_consensus.cc:2229] T 7f616d23efed46418c8f1a9455ce5f3e P fb03b8b8b56e41fb8d81c6d96d959772 [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:13:22.166457 17456 raft_consensus.cc:2258] T 7f616d23efed46418c8f1a9455ce5f3e P fb03b8b8b56e41fb8d81c6d96d959772 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:13:22.168602 17456 ts_tablet_manager.cc:1905] T 7f616d23efed46418c8f1a9455ce5f3e P fb03b8b8b56e41fb8d81c6d96d959772: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:13:22.171487 17432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:22.156890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35528 (local address 127.12.8.1:39793)
0925 15:13:22.157179 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:22.157188 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:22.168277 (+ 11089us) server_negotiation.cc:407] Connection header received
0925 15:13:22.168772 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:22.168780 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:22.168913 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:22.169004 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:22.170381 (+ 1377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:22.170390 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:22.170398 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:13:22.170423 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:22.170471 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:13:22.170492 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:22.170626 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:22.170964 (+ 338us) server_negotiation.cc:299] Negotiation successful
0925 15:13:22.171040 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 15:13:22.172681 17151 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:13:22.172904 17457 tablet_replica.cc:330] T 7f616d23efed46418c8f1a9455ce5f3e P 591bfab4e4054c12bb6d35f58786e750: stopping tablet replica
I20260925 15:13:22.173326 17457 raft_consensus.cc:2229] T 7f616d23efed46418c8f1a9455ce5f3e P 591bfab4e4054c12bb6d35f58786e750 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:13:22.173813 17457 raft_consensus.cc:2258] T 7f616d23efed46418c8f1a9455ce5f3e P 591bfab4e4054c12bb6d35f58786e750 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:13:22.176986 17456 ts_tablet_manager.cc:1918] T 7f616d23efed46418c8f1a9455ce5f3e P fb03b8b8b56e41fb8d81c6d96d959772: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:13:22.177206 17456 log.cc:1196] T 7f616d23efed46418c8f1a9455ce5f3e P fb03b8b8b56e41fb8d81c6d96d959772: Deleting WAL directory at /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349162370577-12320-0/minicluster-data/ts-2/wal/wals/7f616d23efed46418c8f1a9455ce5f3e
I20260925 15:13:22.177785 17456 ts_tablet_manager.cc:1939] T 7f616d23efed46418c8f1a9455ce5f3e P fb03b8b8b56e41fb8d81c6d96d959772: Deleting consensus metadata
I20260925 15:13:22.180032 16954 catalog_manager.cc:4841] TS fb03b8b8b56e41fb8d81c6d96d959772 (127.12.8.3:34067): tablet 7f616d23efed46418c8f1a9455ce5f3e (table test-table [id=ab20debf03754f459a9b731a00a48676]) successfully deleted
I20260925 15:13:22.181767 17457 ts_tablet_manager.cc:1905] T 7f616d23efed46418c8f1a9455ce5f3e P 591bfab4e4054c12bb6d35f58786e750: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:13:22.188519 17457 ts_tablet_manager.cc:1918] T 7f616d23efed46418c8f1a9455ce5f3e P 591bfab4e4054c12bb6d35f58786e750: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:13:22.188678 17457 log.cc:1196] T 7f616d23efed46418c8f1a9455ce5f3e P 591bfab4e4054c12bb6d35f58786e750: Deleting WAL directory at /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349162370577-12320-0/minicluster-data/ts-1/wal/wals/7f616d23efed46418c8f1a9455ce5f3e
I20260925 15:13:22.189221 17457 ts_tablet_manager.cc:1939] T 7f616d23efed46418c8f1a9455ce5f3e P 591bfab4e4054c12bb6d35f58786e750: Deleting consensus metadata
I20260925 15:13:22.190954 16954 catalog_manager.cc:4841] TS 591bfab4e4054c12bb6d35f58786e750 (127.12.8.2:37611): tablet 7f616d23efed46418c8f1a9455ce5f3e (table test-table [id=ab20debf03754f459a9b731a00a48676]) successfully deleted
I20260925 15:13:22.266932 17458 tablet_replica.cc:330] T 7f616d23efed46418c8f1a9455ce5f3e P 4f2a34e3a4654bce9149a2a3b9506162: stopping tablet replica
I20260925 15:13:22.267210 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 17029
I20260925 15:13:22.290033 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 17159
W20260925 15:13:22.290072 16957 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.8.1:39793 (error 108)
I20260925 15:13:22.293494 17433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:22.291225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43644 (local address 127.12.8.2:37611)
0925 15:13:22.291456 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:22.291465 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:22.291492 (+ 27us) server_negotiation.cc:407] Connection header received
0925 15:13:22.291577 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:22.291585 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:22.291723 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:22.291871 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:22.292836 (+ 965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:22.292842 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:22.292847 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:13:22.292867 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:22.292902 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:13:22.292914 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:22.293013 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:22.293153 (+ 140us) server_negotiation.cc:299] Negotiation successful
0925 15:13:22.293213 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 15:13:22.293793 17430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:22.293261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.1:39793 (local address 127.0.0.1:35542)
0925 15:13:22.293510 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:22.293644 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.1:39793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
W20260925 15:13:22.294032 16957 catalog_manager.cc:4584] TS 4f2a34e3a4654bce9149a2a3b9506162 (127.12.8.1:39793): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7f616d23efed46418c8f1a9455ce5f3e: Network error: Client connection negotiation failed: client connection to 127.12.8.1:39793: connect: Connection refused (error 111)
I20260925 15:13:22.294577 17281 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:13:22.347643 17430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:22.347143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.1:39793 (local address 127.0.0.1:35544)
0925 15:13:22.347355 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:22.347464 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.1:39793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260925 15:13:22.348780 17430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:22.348375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.1:39793 (local address 127.0.0.1:35558)
0925 15:13:22.348539 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:22.348642 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.1:39793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260925 15:13:22.381762 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 17159
I20260925 15:13:22.397665 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 17289
I20260925 15:13:22.400828 17429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:22.398483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57122 (local address 127.12.8.3:34067)
0925 15:13:22.398934 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:22.398939 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:22.398953 (+ 14us) server_negotiation.cc:407] Connection header received
0925 15:13:22.399031 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:22.399035 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:22.399080 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:22.399156 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:22.400128 (+ 972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:22.400132 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:22.400135 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:13:22.400153 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:22.400182 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:13:22.400193 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:22.400308 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:22.400424 (+ 116us) server_negotiation.cc:299] Negotiation successful
0925 15:13:22.400471 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347}
I20260925 15:13:22.401440 17411 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:13:22.409431 17430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:22.408928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.1:39793 (local address 127.0.0.1:35572)
0925 15:13:22.409137 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:22.409251 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.1:39793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260925 15:13:22.410611 17430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:22.410083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.1:39793 (local address 127.0.0.1:35588)
0925 15:13:22.410369 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:22.410456 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.1:39793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260925 15:13:22.491261 17430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:22.490715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.1:39793 (local address 127.0.0.1:35596)
0925 15:13:22.490950 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:22.491069 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.1:39793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260925 15:13:22.492462 17430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:22.492015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.1:39793 (local address 127.0.0.1:35602)
0925 15:13:22.492240 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:22.492322 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.1:39793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260925 15:13:22.519165 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 17289
I20260925 15:13:22.537220 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 16939
I20260925 15:13:22.538473 17000 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:13:22.643115 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 16939
2026-09-25T15:13:22Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2505 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260925 15:13:22.683355 12320 mini_oidc.cc:152] Starting JWKS server
I20260925 15:13:22.685846 12320 webserver.cc:460] Webserver started at https://127.0.0.1:43863/ using document root <none> and password file <none>
I20260925 15:13:22.686111 12320 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 15:13:22.686807 12320 webserver.cc:460] Webserver started at http://127.0.0.1:39747/ using document root <none> and password file <none>
2026-09-25T15:13:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:13:22Z Disabled control of system clock
I20260925 15:13:22.711584 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790349162370577-12320-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790349162370577-12320-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790349162370577-12320-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.62:36023
--webserver_interface=127.12.8.62
--webserver_port=0
--builtin_ntp_servers=127.12.8.20:45799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.62:36023
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43863/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790349162370577-12320-0/testchainca.cert with env {}
W20260925 15:13:23.023290 17470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:13:23.023626 17470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:13:23.023779 17470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:13:23.030925 17470 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:13:23.031052 17470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:13:23.031147 17470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:13:23.031271 17470 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:13:23.031352 17470 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 15:13:23.031440 17470 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 15:13:23.031518 17470 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43863/jwks.json
I20260925 15:13:23.041213 17470 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:45799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790349162370577-12320-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790349162370577-12320-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.8.62:36023
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.62:36023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43863/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.8.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790349162370577-12320-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790349162370577-12320-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:13:23.042472 17470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:13:23.043738 17470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:13:23.054842 17475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:23.054925 17476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:23.054896 17478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:13:23.056072 17470 server_base.cc:1034] running on GCE node
I20260925 15:13:23.056851 17470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:13:23.058147 17470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:13:23.059404 17470 hybrid_clock.cc:648] HybridClock initialized: now 1790349203059359 us; error 39 us; skew 500 ppm
I20260925 15:13:23.059736 17470 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:13:23.061872 17470 webserver.cc:460] Webserver started at http://127.12.8.62:36425/ using document root <none> and password file <none>
I20260925 15:13:23.062599 17470 fs_manager.cc:359] Metadata directory not provided
I20260925 15:13:23.062731 17470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:13:23.063027 17470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:13:23.065711 17470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790349162370577-12320-0/minicluster-data/master-0/data/instance:
uuid: "55f4708706624d6abc64f01fab171340"
format_stamp: "Formatted at 2026-09-25 15:13:23 on dist-test-slave-p468"
server_key: "41f7bc00ff8910b20e79d0ebaed4842f"
server_key_iv: "629764816788c2eef9ef6f62723e98f2"
server_key_version: "encryptionkey@0"
I20260925 15:13:23.066550 17470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790349162370577-12320-0/minicluster-data/master-0/wal/instance:
uuid: "55f4708706624d6abc64f01fab171340"
format_stamp: "Formatted at 2026-09-25 15:13:23 on dist-test-slave-p468"
server_key: "41f7bc00ff8910b20e79d0ebaed4842f"
server_key_iv: "629764816788c2eef9ef6f62723e98f2"
server_key_version: "encryptionkey@0"
I20260925 15:13:23.071326 17470 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 15:13:23.074466 17484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:23.075704 17470 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 15:13:23.075884 17470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790349162370577-12320-0/minicluster-data/master-0/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790349162370577-12320-0/minicluster-data/master-0/wal
uuid: "55f4708706624d6abc64f01fab171340"
format_stamp: "Formatted at 2026-09-25 15:13:23 on dist-test-slave-p468"
server_key: "41f7bc00ff8910b20e79d0ebaed4842f"
server_key_iv: "629764816788c2eef9ef6f62723e98f2"
server_key_version: "encryptionkey@0"
I20260925 15:13:23.076082 17470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790349162370577-12320-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790349162370577-12320-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790349162370577-12320-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:13:23.094861 17470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:13:23.095605 17470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:13:23.095870 17470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:13:23.114281 17470 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.62:36023
I20260925 15:13:23.114316 17535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.62:36023 every 8 connection(s)
I20260925 15:13:23.115630 17470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
I20260925 15:13:23.117909 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 17470
I20260925 15:13:23.118115 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790349162370577-12320-0/minicluster-data/master-0/wal/instance
I20260925 15:13:23.118770 12320 external_mini_cluster.cc:1428] Setting key 6bdd962ad5a33a982453fac184feae05
I20260925 15:13:23.119462 17536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:13:23.132875 17536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55f4708706624d6abc64f01fab171340: Bootstrap starting.
I20260925 15:13:23.134315 17539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:23.120598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36670 (local address 127.12.8.62:36023)
0925 15:13:23.121636 (+ 1038us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:23.121650 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:23.121710 (+ 60us) server_negotiation.cc:407] Connection header received
0925 15:13:23.125048 (+ 3338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:23.125081 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:23.125744 (+ 663us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:23.126265 (+ 521us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:23.127324 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:23.128571 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:23.129894 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:23.130653 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:23.130871 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:23.132033 (+ 1162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:23.132040 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:23.132053 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:13:23.132556 (+ 503us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:23.132626 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:13:23.132646 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:23.132769 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:23.132980 (+ 211us) server_negotiation.cc:299] Negotiation successful
0925 15:13:23.133316 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":328,"threads_started":1}
I20260925 15:13:23.136405 17536 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 55f4708706624d6abc64f01fab171340: Neither blocks nor log segments found. Creating new log.
I20260925 15:13:23.137554 17536 log.cc:824] T 00000000000000000000000000000000 P 55f4708706624d6abc64f01fab171340: Log is configured to *not* fsync() on all Append() calls
I20260925 15:13:23.138077 12320 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:13:23.140192 17536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55f4708706624d6abc64f01fab171340: No bootstrap required, opened a new log
I20260925 15:13:23.147616 17536 raft_consensus.cc:348] T 00000000000000000000000000000000 P 55f4708706624d6abc64f01fab171340 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55f4708706624d6abc64f01fab171340" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 36023 } }
I20260925 15:13:23.148010 17536 raft_consensus.cc:374] T 00000000000000000000000000000000 P 55f4708706624d6abc64f01fab171340 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:13:23.148128 17536 raft_consensus.cc:729] T 00000000000000000000000000000000 P 55f4708706624d6abc64f01fab171340 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55f4708706624d6abc64f01fab171340, State: Initialized, Role: FOLLOWER
I20260925 15:13:23.148728 17536 consensus_queue.cc:260] T 00000000000000000000000000000000 P 55f4708706624d6abc64f01fab171340 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55f4708706624d6abc64f01fab171340" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 36023 } }
I20260925 15:13:23.148958 17536 raft_consensus.cc:388] T 00000000000000000000000000000000 P 55f4708706624d6abc64f01fab171340 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:13:23.149087 17536 raft_consensus.cc:482] T 00000000000000000000000000000000 P 55f4708706624d6abc64f01fab171340 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:13:23.149289 17536 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 55f4708706624d6abc64f01fab171340 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:13:23.151393 17536 raft_consensus.cc:504] T 00000000000000000000000000000000 P 55f4708706624d6abc64f01fab171340 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55f4708706624d6abc64f01fab171340" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 36023 } }
I20260925 15:13:23.151916 17536 leader_election.cc:302] T 00000000000000000000000000000000 P 55f4708706624d6abc64f01fab171340 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 55f4708706624d6abc64f01fab171340; no voters:
I20260925 15:13:23.152482 17536 leader_election.cc:288] T 00000000000000000000000000000000 P 55f4708706624d6abc64f01fab171340 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:13:23.152622 17541 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 55f4708706624d6abc64f01fab171340 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:13:23.153642 17541 raft_consensus.cc:686] T 00000000000000000000000000000000 P 55f4708706624d6abc64f01fab171340 [term 1 LEADER]: Becoming Leader. State: Replica: 55f4708706624d6abc64f01fab171340, State: Running, Role: LEADER
I20260925 15:13:23.154120 17541 consensus_queue.cc:237] T 00000000000000000000000000000000 P 55f4708706624d6abc64f01fab171340 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55f4708706624d6abc64f01fab171340" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 36023 } }
I20260925 15:13:23.154634 17536 sys_catalog.cc:564] T 00000000000000000000000000000000 P 55f4708706624d6abc64f01fab171340 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:13:23.156857 17542 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55f4708706624d6abc64f01fab171340 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "55f4708706624d6abc64f01fab171340" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55f4708706624d6abc64f01fab171340" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 36023 } } }
I20260925 15:13:23.157258 17542 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55f4708706624d6abc64f01fab171340 [sys.catalog]: This master's current role is: LEADER
I20260925 15:13:23.157662 17543 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55f4708706624d6abc64f01fab171340 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 55f4708706624d6abc64f01fab171340. Latest consensus state: current_term: 1 leader_uuid: "55f4708706624d6abc64f01fab171340" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55f4708706624d6abc64f01fab171340" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 36023 } } }
I20260925 15:13:23.157963 17543 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55f4708706624d6abc64f01fab171340 [sys.catalog]: This master's current role is: LEADER
I20260925 15:13:23.158231 17550 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:13:23.164264 17550 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:13:23.172123 17550 catalog_manager.cc:1347] Generated new cluster ID: 3eaaaf26f877472cb039a105636dae69
I20260925 15:13:23.172221 17550 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:13:23.204603 17550 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:13:23.205871 17550 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:13:23.216894 17550 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 55f4708706624d6abc64f01fab171340: Generated new TSK 0
I20260925 15:13:23.217690 17550 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 15:13:23.264844 17566 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260925 15:13:23.265025 17566 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 15:13:23.266180 17539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:23.244399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36672 (local address 127.12.8.62:36023)
0925 15:13:23.244597 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:23.244603 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:23.244993 (+ 390us) server_negotiation.cc:407] Connection header received
0925 15:13:23.245252 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:23.245258 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:23.245344 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:23.245508 (+ 164us) server_negotiation.cc:226] Negotiated authn=JWT
0925 15:13:23.246043 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:23.247202 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:23.248591 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:23.249307 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:23.249433 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:23.250078 (+ 645us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 15:13:23.265509 (+ 15431us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43863/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 15:13:23.265872 (+ 363us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36672: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43863/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":67}
W20260925 15:13:23.266372 17539 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36672: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43863/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260925 15:13:23.267539 17565 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:13:23.244265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:36023 (local address 127.0.0.1:36672)
0925 15:13:23.244892 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:23.244908 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:23.245049 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:23.245543 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:23.245547 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:23.245557 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0925 15:13:23.245892 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:23.245901 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:23.247342 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:23.247346 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:23.248459 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:23.248469 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:23.249460 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:23.249528 (+ 68us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:23.249728 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:23.249814 (+ 86us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 15:13:23.265785 (+ 15971us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43863/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 15:13:23.266043 (+ 258us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.12.8.62:36023: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43863/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":323,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.12.8.62:36023: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43863/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260925 15:13:23.269683 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 17470
I20260925 15:13:23.270792 17531 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:13:23.364467 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 17470
2026-09-25T15:13:23Z chronyd exiting
I20260925 15:13:23.410570 12320 test_util.cc:175] -----------------------------------------------
I20260925 15:13:23.410744 12320 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790349162370577-12320-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (731 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260925 15:13:23.414389 12320 mini_oidc.cc:152] Starting JWKS server
I20260925 15:13:23.416208 12320 webserver.cc:460] Webserver started at https://127.0.0.1:37397/ using document root <none> and password file <none>
I20260925 15:13:23.416442 12320 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 15:13:23.417078 12320 webserver.cc:460] Webserver started at http://127.0.0.1:37515/ using document root <none> and password file <none>
2026-09-25T15:13:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:13:23Z Disabled control of system clock
I20260925 15:13:23.442001 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790349162370577-12320-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790349162370577-12320-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790349162370577-12320-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.62:35537
--webserver_interface=127.12.8.62
--webserver_port=0
--builtin_ntp_servers=127.12.8.20:37027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.62:35537
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37397/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790349162370577-12320-0/testchainca.cert with env {}
W20260925 15:13:23.748760 17573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:13:23.749024 17573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:13:23.749112 17573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:13:23.756130 17573 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:13:23.756237 17573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:13:23.756284 17573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:13:23.756335 17573 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:13:23.756379 17573 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 15:13:23.756434 17573 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 15:13:23.756474 17573 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37397/jwks.json
I20260925 15:13:23.765872 17573 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:37027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790349162370577-12320-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790349162370577-12320-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.8.62:35537
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.62:35537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37397/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.8.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790349162370577-12320-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790349162370577-12320-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:13:23.767117 17573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:13:23.768363 17573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:13:23.782585 17578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:23.782639 17579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:23.782610 17581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:13:23.783919 17573 server_base.cc:1034] running on GCE node
I20260925 15:13:23.784682 17573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:13:23.786072 17573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:13:23.787341 17573 hybrid_clock.cc:648] HybridClock initialized: now 1790349203787280 us; error 61 us; skew 500 ppm
I20260925 15:13:23.787707 17573 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:13:23.795363 17573 webserver.cc:460] Webserver started at http://127.12.8.62:41527/ using document root <none> and password file <none>
I20260925 15:13:23.796074 17573 fs_manager.cc:359] Metadata directory not provided
I20260925 15:13:23.796222 17573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:13:23.796547 17573 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:13:23.799448 17573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790349162370577-12320-0/minicluster-data/master-0/data/instance:
uuid: "c3d637f3f0dd408aabf6e5363c8934e3"
format_stamp: "Formatted at 2026-09-25 15:13:23 on dist-test-slave-p468"
server_key: "49879f0d06de1e9fcd7eb8eecc90db29"
server_key_iv: "034ac3a7c52a02ca399a7ca0c717f324"
server_key_version: "encryptionkey@0"
I20260925 15:13:23.800256 17573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790349162370577-12320-0/minicluster-data/master-0/wal/instance:
uuid: "c3d637f3f0dd408aabf6e5363c8934e3"
format_stamp: "Formatted at 2026-09-25 15:13:23 on dist-test-slave-p468"
server_key: "49879f0d06de1e9fcd7eb8eecc90db29"
server_key_iv: "034ac3a7c52a02ca399a7ca0c717f324"
server_key_version: "encryptionkey@0"
I20260925 15:13:23.805279 17573 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260925 15:13:23.808727 17587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:23.809973 17573 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 15:13:23.810174 17573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790349162370577-12320-0/minicluster-data/master-0/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790349162370577-12320-0/minicluster-data/master-0/wal
uuid: "c3d637f3f0dd408aabf6e5363c8934e3"
format_stamp: "Formatted at 2026-09-25 15:13:23 on dist-test-slave-p468"
server_key: "49879f0d06de1e9fcd7eb8eecc90db29"
server_key_iv: "034ac3a7c52a02ca399a7ca0c717f324"
server_key_version: "encryptionkey@0"
I20260925 15:13:23.810421 17573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790349162370577-12320-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790349162370577-12320-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790349162370577-12320-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:13:23.831857 17573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:13:23.832680 17573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:13:23.832965 17573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:13:23.851420 17573 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.62:35537
I20260925 15:13:23.851440 17638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.62:35537 every 8 connection(s)
I20260925 15:13:23.852747 17573 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
I20260925 15:13:23.856653 17639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:13:23.857858 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 17573
I20260925 15:13:23.858088 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790349162370577-12320-0/minicluster-data/master-0/wal/instance
I20260925 15:13:23.858999 12320 external_mini_cluster.cc:1428] Setting key 63adb5272cf434b5e75492c4e6baf103
I20260925 15:13:23.874151 17639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3d637f3f0dd408aabf6e5363c8934e3: Bootstrap starting.
I20260925 15:13:23.878854 17639 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c3d637f3f0dd408aabf6e5363c8934e3: Neither blocks nor log segments found. Creating new log.
I20260925 15:13:23.880394 17639 log.cc:824] T 00000000000000000000000000000000 P c3d637f3f0dd408aabf6e5363c8934e3: Log is configured to *not* fsync() on all Append() calls
I20260925 15:13:23.883620 17639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3d637f3f0dd408aabf6e5363c8934e3: No bootstrap required, opened a new log
I20260925 15:13:23.887620 17642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:23.866851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53056 (local address 127.12.8.62:35537)
0925 15:13:23.874433 (+ 7582us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:23.874448 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:23.874495 (+ 47us) server_negotiation.cc:407] Connection header received
0925 15:13:23.876892 (+ 2397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:23.876922 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:23.877560 (+ 638us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:23.877976 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:23.878631 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:23.880682 (+ 2051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:23.882748 (+ 2066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:23.883455 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:23.883656 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:23.885152 (+ 1496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:23.885160 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:23.885173 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:13:23.885766 (+ 593us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:23.885849 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:13:23.885870 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:23.886031 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:23.886267 (+ 236us) server_negotiation.cc:299] Negotiation successful
0925 15:13:23.886628 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7030,"thread_start_us":7327,"threads_started":1}
I20260925 15:13:23.891523 12320 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:13:23.891878 17639 raft_consensus.cc:348] T 00000000000000000000000000000000 P c3d637f3f0dd408aabf6e5363c8934e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3d637f3f0dd408aabf6e5363c8934e3" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 35537 } }
I20260925 15:13:23.892225 17639 raft_consensus.cc:374] T 00000000000000000000000000000000 P c3d637f3f0dd408aabf6e5363c8934e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:13:23.892341 17639 raft_consensus.cc:729] T 00000000000000000000000000000000 P c3d637f3f0dd408aabf6e5363c8934e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3d637f3f0dd408aabf6e5363c8934e3, State: Initialized, Role: FOLLOWER
I20260925 15:13:23.892858 17639 consensus_queue.cc:260] T 00000000000000000000000000000000 P c3d637f3f0dd408aabf6e5363c8934e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3d637f3f0dd408aabf6e5363c8934e3" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 35537 } }
I20260925 15:13:23.893095 17639 raft_consensus.cc:388] T 00000000000000000000000000000000 P c3d637f3f0dd408aabf6e5363c8934e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:13:23.893244 17639 raft_consensus.cc:482] T 00000000000000000000000000000000 P c3d637f3f0dd408aabf6e5363c8934e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:13:23.893421 17639 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c3d637f3f0dd408aabf6e5363c8934e3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:13:23.896356 17639 raft_consensus.cc:504] T 00000000000000000000000000000000 P c3d637f3f0dd408aabf6e5363c8934e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3d637f3f0dd408aabf6e5363c8934e3" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 35537 } }
I20260925 15:13:23.896880 17639 leader_election.cc:302] T 00000000000000000000000000000000 P c3d637f3f0dd408aabf6e5363c8934e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c3d637f3f0dd408aabf6e5363c8934e3; no voters:
I20260925 15:13:23.897543 17639 leader_election.cc:288] T 00000000000000000000000000000000 P c3d637f3f0dd408aabf6e5363c8934e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:13:23.897691 17644 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c3d637f3f0dd408aabf6e5363c8934e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:13:23.898415 17644 raft_consensus.cc:686] T 00000000000000000000000000000000 P c3d637f3f0dd408aabf6e5363c8934e3 [term 1 LEADER]: Becoming Leader. State: Replica: c3d637f3f0dd408aabf6e5363c8934e3, State: Running, Role: LEADER
I20260925 15:13:23.898937 17644 consensus_queue.cc:237] T 00000000000000000000000000000000 P c3d637f3f0dd408aabf6e5363c8934e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3d637f3f0dd408aabf6e5363c8934e3" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 35537 } }
I20260925 15:13:23.899554 17639 sys_catalog.cc:564] T 00000000000000000000000000000000 P c3d637f3f0dd408aabf6e5363c8934e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:13:23.901777 17646 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3d637f3f0dd408aabf6e5363c8934e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3d637f3f0dd408aabf6e5363c8934e3. Latest consensus state: current_term: 1 leader_uuid: "c3d637f3f0dd408aabf6e5363c8934e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3d637f3f0dd408aabf6e5363c8934e3" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 35537 } } }
I20260925 15:13:23.901746 17645 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3d637f3f0dd408aabf6e5363c8934e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c3d637f3f0dd408aabf6e5363c8934e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3d637f3f0dd408aabf6e5363c8934e3" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 35537 } } }
I20260925 15:13:23.902179 17646 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3d637f3f0dd408aabf6e5363c8934e3 [sys.catalog]: This master's current role is: LEADER
I20260925 15:13:23.902245 17645 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3d637f3f0dd408aabf6e5363c8934e3 [sys.catalog]: This master's current role is: LEADER
I20260925 15:13:23.903733 17649 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:13:23.907946 17649 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:13:23.915895 17649 catalog_manager.cc:1347] Generated new cluster ID: a7aa9dae199e49e3941dc2c95d4c97d2
I20260925 15:13:23.915997 17649 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:13:23.942984 17649 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:13:23.944890 17649 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:13:23.961448 17649 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c3d637f3f0dd408aabf6e5363c8934e3: Generated new TSK 0
I20260925 15:13:23.962587 17649 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 15:13:24.016259 17669 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260925 15:13:24.016433 17669 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 15:13:24.018103 17642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:23.984761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53064 (local address 127.12.8.62:35537)
0925 15:13:23.986368 (+ 1607us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:23.986375 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:23.986947 (+ 572us) server_negotiation.cc:407] Connection header received
0925 15:13:23.987249 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:23.987257 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:23.987356 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:23.987504 (+ 148us) server_negotiation.cc:226] Negotiated authn=JWT
0925 15:13:23.988247 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:23.989324 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:23.990964 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:23.991395 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:23.991526 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:23.992014 (+ 488us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 15:13:24.017271 (+ 25257us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37397/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 15:13:24.017750 (+ 479us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53064: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37397/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":1466}
W20260925 15:13:24.018342 17642 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53064: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37397/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260925 15:13:24.019078 17668 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:13:23.984626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:35537 (local address 127.0.0.1:53064)
0925 15:13:23.986835 (+ 2209us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:23.986863 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:23.987063 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:23.987687 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:23.987696 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:23.987714 (+ 18us) client_negotiation.cc:189] Negotiated authn=JWT
0925 15:13:23.988023 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:23.988034 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:23.989514 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:23.989521 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:23.990811 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:23.990823 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:23.991514 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:23.991560 (+ 46us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:23.991721 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:23.991753 (+ 32us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 15:13:24.018382 (+ 26629us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37397/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 15:13:24.018667 (+ 285us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.12.8.62:35537: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37397/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":2052,"thread_start_us":1897,"threads_started":1}
I20260925 15:13:24.020241 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 17573
I20260925 15:13:24.021608 17634 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:13:24.114053 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 17573
2026-09-25T15:13:24Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (754 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260925 15:13:24.169054 12320 mini_oidc.cc:152] Starting JWKS server
I20260925 15:13:24.170929 12320 webserver.cc:460] Webserver started at https://127.0.0.1:36617/ using document root <none> and password file <none>
I20260925 15:13:24.171129 12320 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 15:13:24.171659 12320 webserver.cc:460] Webserver started at http://127.0.0.1:41963/ using document root <none> and password file <none>
2026-09-25T15:13:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:13:24Z Disabled control of system clock
I20260925 15:13:24.196441 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790349162370577-12320-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790349162370577-12320-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790349162370577-12320-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.62:43883
--webserver_interface=127.12.8.62
--webserver_port=0
--builtin_ntp_servers=127.12.8.20:38817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.62:43883
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36617/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 15:13:24.502652 17676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:13:24.502926 17676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:13:24.503120 17676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:13:24.510762 17676 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:13:24.510882 17676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:13:24.510951 17676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:13:24.511032 17676 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:13:24.511094 17676 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 15:13:24.511184 17676 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 15:13:24.511245 17676 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36617/jwks.json
I20260925 15:13:24.520330 17676 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:38817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790349162370577-12320-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790349162370577-12320-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.8.62:43883
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.62:43883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36617/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.8.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790349162370577-12320-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:13:24.521564 17676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:13:24.523330 17676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:13:24.534595 17681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:24.534626 17684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:24.534675 17682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:13:24.536062 17676 server_base.cc:1034] running on GCE node
I20260925 15:13:24.536823 17676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:13:24.538149 17676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:13:24.539397 17676 hybrid_clock.cc:648] HybridClock initialized: now 1790349204539358 us; error 44 us; skew 500 ppm
I20260925 15:13:24.539748 17676 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:13:24.541915 17676 webserver.cc:460] Webserver started at http://127.12.8.62:36279/ using document root <none> and password file <none>
I20260925 15:13:24.542644 17676 fs_manager.cc:359] Metadata directory not provided
I20260925 15:13:24.542776 17676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:13:24.543087 17676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:13:24.545768 17676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790349162370577-12320-0/minicluster-data/master-0/data/instance:
uuid: "898df6cd73c44beebc3e574cebbdf612"
format_stamp: "Formatted at 2026-09-25 15:13:24 on dist-test-slave-p468"
server_key: "758fca4d8d579e6581f395cfdbb176f7"
server_key_iv: "9228ed163042e95943b3ec67f6e27ea2"
server_key_version: "encryptionkey@0"
I20260925 15:13:24.546586 17676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790349162370577-12320-0/minicluster-data/master-0/wal/instance:
uuid: "898df6cd73c44beebc3e574cebbdf612"
format_stamp: "Formatted at 2026-09-25 15:13:24 on dist-test-slave-p468"
server_key: "758fca4d8d579e6581f395cfdbb176f7"
server_key_iv: "9228ed163042e95943b3ec67f6e27ea2"
server_key_version: "encryptionkey@0"
I20260925 15:13:24.551375 17676 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 15:13:24.554467 17690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:24.555681 17676 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 15:13:24.555858 17676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790349162370577-12320-0/minicluster-data/master-0/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790349162370577-12320-0/minicluster-data/master-0/wal
uuid: "898df6cd73c44beebc3e574cebbdf612"
format_stamp: "Formatted at 2026-09-25 15:13:24 on dist-test-slave-p468"
server_key: "758fca4d8d579e6581f395cfdbb176f7"
server_key_iv: "9228ed163042e95943b3ec67f6e27ea2"
server_key_version: "encryptionkey@0"
I20260925 15:13:24.556061 17676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790349162370577-12320-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790349162370577-12320-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790349162370577-12320-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:13:24.579813 17676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:13:24.580598 17676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:13:24.580880 17676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:13:24.598553 17676 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.62:43883
I20260925 15:13:24.598572 17741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.62:43883 every 8 connection(s)
I20260925 15:13:24.599815 17676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
I20260925 15:13:24.602424 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 17676
I20260925 15:13:24.602640 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790349162370577-12320-0/minicluster-data/master-0/wal/instance
I20260925 15:13:24.603400 12320 external_mini_cluster.cc:1428] Setting key 5fa5e067a77db44fabd9bfe5f19b5cdd
I20260925 15:13:24.603928 17742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:13:24.616676 17745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:24.605177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50386 (local address 127.12.8.62:43883)
0925 15:13:24.606114 (+ 937us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:24.606124 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:24.606161 (+ 37us) server_negotiation.cc:407] Connection header received
0925 15:13:24.608226 (+ 2065us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:24.608246 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:24.608666 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:24.608967 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:24.609501 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:24.610844 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:24.612590 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:24.613191 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:24.613336 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:24.614514 (+ 1178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:24.614522 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:24.614533 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:13:24.615069 (+ 536us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:24.615142 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:13:24.615160 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:24.615258 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:24.615480 (+ 222us) server_negotiation.cc:299] Negotiation successful
0925 15:13:24.615792 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":369,"threads_started":1}
I20260925 15:13:24.617084 17742 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 898df6cd73c44beebc3e574cebbdf612: Bootstrap starting.
I20260925 15:13:24.620290 17742 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 898df6cd73c44beebc3e574cebbdf612: Neither blocks nor log segments found. Creating new log.
I20260925 15:13:24.620602 12320 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:13:24.621299 17742 log.cc:824] T 00000000000000000000000000000000 P 898df6cd73c44beebc3e574cebbdf612: Log is configured to *not* fsync() on all Append() calls
I20260925 15:13:24.624140 17742 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 898df6cd73c44beebc3e574cebbdf612: No bootstrap required, opened a new log
I20260925 15:13:24.631373 17742 raft_consensus.cc:348] T 00000000000000000000000000000000 P 898df6cd73c44beebc3e574cebbdf612 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "898df6cd73c44beebc3e574cebbdf612" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 43883 } }
I20260925 15:13:24.631707 17742 raft_consensus.cc:374] T 00000000000000000000000000000000 P 898df6cd73c44beebc3e574cebbdf612 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:13:24.631831 17742 raft_consensus.cc:729] T 00000000000000000000000000000000 P 898df6cd73c44beebc3e574cebbdf612 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 898df6cd73c44beebc3e574cebbdf612, State: Initialized, Role: FOLLOWER
I20260925 15:13:24.632342 17742 consensus_queue.cc:260] T 00000000000000000000000000000000 P 898df6cd73c44beebc3e574cebbdf612 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "898df6cd73c44beebc3e574cebbdf612" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 43883 } }
I20260925 15:13:24.632604 17742 raft_consensus.cc:388] T 00000000000000000000000000000000 P 898df6cd73c44beebc3e574cebbdf612 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:13:24.632776 17742 raft_consensus.cc:482] T 00000000000000000000000000000000 P 898df6cd73c44beebc3e574cebbdf612 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:13:24.632958 17742 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 898df6cd73c44beebc3e574cebbdf612 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:13:24.635699 17742 raft_consensus.cc:504] T 00000000000000000000000000000000 P 898df6cd73c44beebc3e574cebbdf612 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "898df6cd73c44beebc3e574cebbdf612" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 43883 } }
I20260925 15:13:24.636175 17742 leader_election.cc:302] T 00000000000000000000000000000000 P 898df6cd73c44beebc3e574cebbdf612 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 898df6cd73c44beebc3e574cebbdf612; no voters:
I20260925 15:13:24.636771 17742 leader_election.cc:288] T 00000000000000000000000000000000 P 898df6cd73c44beebc3e574cebbdf612 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:13:24.636868 17747 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 898df6cd73c44beebc3e574cebbdf612 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:13:24.637606 17747 raft_consensus.cc:686] T 00000000000000000000000000000000 P 898df6cd73c44beebc3e574cebbdf612 [term 1 LEADER]: Becoming Leader. State: Replica: 898df6cd73c44beebc3e574cebbdf612, State: Running, Role: LEADER
I20260925 15:13:24.638181 17747 consensus_queue.cc:237] T 00000000000000000000000000000000 P 898df6cd73c44beebc3e574cebbdf612 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "898df6cd73c44beebc3e574cebbdf612" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 43883 } }
I20260925 15:13:24.638900 17742 sys_catalog.cc:564] T 00000000000000000000000000000000 P 898df6cd73c44beebc3e574cebbdf612 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:13:24.640895 17748 sys_catalog.cc:455] T 00000000000000000000000000000000 P 898df6cd73c44beebc3e574cebbdf612 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "898df6cd73c44beebc3e574cebbdf612" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "898df6cd73c44beebc3e574cebbdf612" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 43883 } } }
I20260925 15:13:24.640908 17749 sys_catalog.cc:455] T 00000000000000000000000000000000 P 898df6cd73c44beebc3e574cebbdf612 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 898df6cd73c44beebc3e574cebbdf612. Latest consensus state: current_term: 1 leader_uuid: "898df6cd73c44beebc3e574cebbdf612" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "898df6cd73c44beebc3e574cebbdf612" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 43883 } } }
I20260925 15:13:24.641345 17748 sys_catalog.cc:458] T 00000000000000000000000000000000 P 898df6cd73c44beebc3e574cebbdf612 [sys.catalog]: This master's current role is: LEADER
I20260925 15:13:24.641345 17749 sys_catalog.cc:458] T 00000000000000000000000000000000 P 898df6cd73c44beebc3e574cebbdf612 [sys.catalog]: This master's current role is: LEADER
I20260925 15:13:24.650213 17753 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:13:24.656399 17753 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:13:24.665042 17753 catalog_manager.cc:1347] Generated new cluster ID: f7b879518147437fa7bb73a9890c3b1f
I20260925 15:13:24.665169 17753 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:13:24.682103 17753 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:13:24.683320 17753 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:13:24.690814 17753 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 898df6cd73c44beebc3e574cebbdf612: Generated new TSK 0
I20260925 15:13:24.691556 17753 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:13:24.713398 17745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:24.705423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50400 (local address 127.12.8.62:43883)
0925 15:13:24.705602 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:24.705608 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:24.706016 (+ 408us) server_negotiation.cc:407] Connection header received
0925 15:13:24.706231 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:24.706237 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:24.706333 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:24.706485 (+ 152us) server_negotiation.cc:226] Negotiated authn=JWT
0925 15:13:24.706883 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:24.707991 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:24.709291 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:24.709977 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:24.710092 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:24.710607 (+ 515us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 15:13:24.712809 (+ 2202us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36617/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 15:13:24.713109 (+ 300us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50400: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36617/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":51}
W20260925 15:13:24.713516 17745 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50400: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36617/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260925 15:13:24.715231 17771 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:13:24.705259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:43883 (local address 127.0.0.1:50400)
0925 15:13:24.705924 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:24.705938 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:24.706048 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:24.706496 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:24.706500 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:24.706509 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0925 15:13:24.706739 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:24.706746 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:24.708118 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:24.708122 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:24.709160 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:24.709170 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:24.710096 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:24.710153 (+ 57us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:24.710329 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:24.710359 (+ 30us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 15:13:24.712998 (+ 2639us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36617/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 15:13:24.714824 (+ 1826us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.12.8.62:43883: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36617/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":542,"mutex_wait_us":1546,"thread_start_us":383,"threads_started":1}
I20260925 15:13:24.716233 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 17676
I20260925 15:13:24.717293 17737 generic_service.cc:196] Checking for leaks (request via RPC)
E20260925 15:13:24.725207 17772 webserver.cc:533] Webserver: sslize failed:
E20260925 15:13:24.725332 17772 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 15:13:24.805868 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 17676
2026-09-25T15:13:24Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (691 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-25T15:13:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:13:24Z Disabled control of system clock
I20260925 15:13:24.884848 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790349162370577-12320-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790349162370577-12320-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790349162370577-12320-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.62:38999
--webserver_interface=127.12.8.62
--webserver_port=0
--builtin_ntp_servers=127.12.8.20:36367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.62:38999
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790349162370577-12320-0/testchainca.cert with env {}
W20260925 15:13:25.195493 17777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:13:25.195787 17777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:13:25.195919 17777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:13:25.203672 17777 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:13:25.203779 17777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:13:25.203857 17777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:13:25.203938 17777 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:13:25.204011 17777 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 15:13:25.204099 17777 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 15:13:25.204166 17777 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260925 15:13:25.213919 17777 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:36367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790349162370577-12320-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790349162370577-12320-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.8.62:38999
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.62:38999
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.8.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790349162370577-12320-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790349162370577-12320-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:13:25.215242 17777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:13:25.216497 17777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:13:25.227939 17782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:25.227988 17783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:25.228008 17785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:13:25.228888 17777 server_base.cc:1034] running on GCE node
I20260925 15:13:25.229619 17777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:13:25.231047 17777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:13:25.232275 17777 hybrid_clock.cc:648] HybridClock initialized: now 1790349205232249 us; error 56 us; skew 500 ppm
I20260925 15:13:25.232621 17777 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:13:25.234863 17777 webserver.cc:460] Webserver started at http://127.12.8.62:46045/ using document root <none> and password file <none>
I20260925 15:13:25.235543 17777 fs_manager.cc:359] Metadata directory not provided
I20260925 15:13:25.235677 17777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:13:25.235973 17777 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:13:25.238788 17777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790349162370577-12320-0/minicluster-data/master-0/data/instance:
uuid: "149fbe5d985b4620ab42811195cb6aac"
format_stamp: "Formatted at 2026-09-25 15:13:25 on dist-test-slave-p468"
server_key: "dc16a7eefd23f94fde1c92c6c75980e0"
server_key_iv: "3707922117c2d822a11eaec10265a91e"
server_key_version: "encryptionkey@0"
I20260925 15:13:25.239558 17777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790349162370577-12320-0/minicluster-data/master-0/wal/instance:
uuid: "149fbe5d985b4620ab42811195cb6aac"
format_stamp: "Formatted at 2026-09-25 15:13:25 on dist-test-slave-p468"
server_key: "dc16a7eefd23f94fde1c92c6c75980e0"
server_key_iv: "3707922117c2d822a11eaec10265a91e"
server_key_version: "encryptionkey@0"
I20260925 15:13:25.244472 17777 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260925 15:13:25.247648 17791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:25.248883 17777 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 15:13:25.249053 17777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790349162370577-12320-0/minicluster-data/master-0/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790349162370577-12320-0/minicluster-data/master-0/wal
uuid: "149fbe5d985b4620ab42811195cb6aac"
format_stamp: "Formatted at 2026-09-25 15:13:25 on dist-test-slave-p468"
server_key: "dc16a7eefd23f94fde1c92c6c75980e0"
server_key_iv: "3707922117c2d822a11eaec10265a91e"
server_key_version: "encryptionkey@0"
I20260925 15:13:25.249269 17777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790349162370577-12320-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790349162370577-12320-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790349162370577-12320-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:13:25.267498 17777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:13:25.268268 17777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:13:25.268543 17777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:13:25.285696 17777 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.62:38999
I20260925 15:13:25.285715 17842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.62:38999 every 8 connection(s)
I20260925 15:13:25.287043 17777 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
I20260925 15:13:25.290721 17843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:13:25.291282 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 17777
I20260925 15:13:25.291481 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790349162370577-12320-0/minicluster-data/master-0/wal/instance
I20260925 15:13:25.292098 12320 external_mini_cluster.cc:1428] Setting key f63c8dc4d709d365f436b8eced73aaca
I20260925 15:13:25.303575 17843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 149fbe5d985b4620ab42811195cb6aac: Bootstrap starting.
I20260925 15:13:25.305207 17846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:25.293857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52612 (local address 127.12.8.62:38999)
0925 15:13:25.294828 (+ 971us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:25.294840 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:25.294880 (+ 40us) server_negotiation.cc:407] Connection header received
0925 15:13:25.296460 (+ 1580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:25.296479 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:25.296910 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:25.297213 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:25.297763 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:25.299036 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:25.300848 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:25.301544 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:25.301680 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:25.303023 (+ 1343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:25.303030 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:25.303044 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:13:25.303559 (+ 515us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:25.303633 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:13:25.303653 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:25.303787 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:25.304020 (+ 233us) server_negotiation.cc:299] Negotiation successful
0925 15:13:25.304327 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":373,"threads_started":1}
I20260925 15:13:25.306982 17843 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 149fbe5d985b4620ab42811195cb6aac: Neither blocks nor log segments found. Creating new log.
I20260925 15:13:25.308128 17843 log.cc:824] T 00000000000000000000000000000000 P 149fbe5d985b4620ab42811195cb6aac: Log is configured to *not* fsync() on all Append() calls
I20260925 15:13:25.308884 12320 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:13:25.311147 17843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 149fbe5d985b4620ab42811195cb6aac: No bootstrap required, opened a new log
I20260925 15:13:25.318251 17843 raft_consensus.cc:348] T 00000000000000000000000000000000 P 149fbe5d985b4620ab42811195cb6aac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "149fbe5d985b4620ab42811195cb6aac" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 38999 } }
I20260925 15:13:25.318656 17843 raft_consensus.cc:374] T 00000000000000000000000000000000 P 149fbe5d985b4620ab42811195cb6aac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:13:25.318765 17843 raft_consensus.cc:729] T 00000000000000000000000000000000 P 149fbe5d985b4620ab42811195cb6aac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149fbe5d985b4620ab42811195cb6aac, State: Initialized, Role: FOLLOWER
I20260925 15:13:25.319355 17843 consensus_queue.cc:260] T 00000000000000000000000000000000 P 149fbe5d985b4620ab42811195cb6aac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "149fbe5d985b4620ab42811195cb6aac" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 38999 } }
I20260925 15:13:25.319579 17843 raft_consensus.cc:388] T 00000000000000000000000000000000 P 149fbe5d985b4620ab42811195cb6aac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:13:25.319723 17843 raft_consensus.cc:482] T 00000000000000000000000000000000 P 149fbe5d985b4620ab42811195cb6aac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:13:25.319893 17843 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 149fbe5d985b4620ab42811195cb6aac [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:13:25.322063 17843 raft_consensus.cc:504] T 00000000000000000000000000000000 P 149fbe5d985b4620ab42811195cb6aac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "149fbe5d985b4620ab42811195cb6aac" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 38999 } }
I20260925 15:13:25.322564 17843 leader_election.cc:302] T 00000000000000000000000000000000 P 149fbe5d985b4620ab42811195cb6aac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 149fbe5d985b4620ab42811195cb6aac; no voters:
I20260925 15:13:25.323213 17843 leader_election.cc:288] T 00000000000000000000000000000000 P 149fbe5d985b4620ab42811195cb6aac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:13:25.323381 17848 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 149fbe5d985b4620ab42811195cb6aac [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:13:25.324105 17848 raft_consensus.cc:686] T 00000000000000000000000000000000 P 149fbe5d985b4620ab42811195cb6aac [term 1 LEADER]: Becoming Leader. State: Replica: 149fbe5d985b4620ab42811195cb6aac, State: Running, Role: LEADER
I20260925 15:13:25.324651 17848 consensus_queue.cc:237] T 00000000000000000000000000000000 P 149fbe5d985b4620ab42811195cb6aac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "149fbe5d985b4620ab42811195cb6aac" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 38999 } }
I20260925 15:13:25.325397 17843 sys_catalog.cc:564] T 00000000000000000000000000000000 P 149fbe5d985b4620ab42811195cb6aac [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:13:25.327524 17849 sys_catalog.cc:455] T 00000000000000000000000000000000 P 149fbe5d985b4620ab42811195cb6aac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "149fbe5d985b4620ab42811195cb6aac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "149fbe5d985b4620ab42811195cb6aac" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 38999 } } }
I20260925 15:13:25.327935 17849 sys_catalog.cc:458] T 00000000000000000000000000000000 P 149fbe5d985b4620ab42811195cb6aac [sys.catalog]: This master's current role is: LEADER
I20260925 15:13:25.328025 17850 sys_catalog.cc:455] T 00000000000000000000000000000000 P 149fbe5d985b4620ab42811195cb6aac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 149fbe5d985b4620ab42811195cb6aac. Latest consensus state: current_term: 1 leader_uuid: "149fbe5d985b4620ab42811195cb6aac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "149fbe5d985b4620ab42811195cb6aac" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 38999 } } }
I20260925 15:13:25.328367 17850 sys_catalog.cc:458] T 00000000000000000000000000000000 P 149fbe5d985b4620ab42811195cb6aac [sys.catalog]: This master's current role is: LEADER
I20260925 15:13:25.330845 17855 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:13:25.335536 17855 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:13:25.344019 17855 catalog_manager.cc:1347] Generated new cluster ID: a17d0e1916884f74b5af08ad3179cc9e
I20260925 15:13:25.344139 17855 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:13:25.358793 17855 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:13:25.360035 17855 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:13:25.373196 17855 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 149fbe5d985b4620ab42811195cb6aac: Generated new TSK 0
I20260925 15:13:25.374076 17855 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:13:25.436319 17846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:25.388204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52622 (local address 127.12.8.62:38999)
0925 15:13:25.388441 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:25.388448 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:25.388822 (+ 374us) server_negotiation.cc:407] Connection header received
0925 15:13:25.389056 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:25.389059 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:25.389127 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:25.389278 (+ 151us) server_negotiation.cc:226] Negotiated authn=JWT
0925 15:13:25.389805 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:25.390632 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:25.392675 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:25.393121 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:25.393243 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:25.393787 (+ 544us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 15:13:25.435745 (+ 41958us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 15:13:25.436067 (+ 322us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52622: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":69}
W20260925 15:13:25.436486 17846 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52622: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260925 15:13:25.436731 17872 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:13:25.388094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:38999 (local address 127.0.0.1:52622)
0925 15:13:25.388695 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:25.388715 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:25.388886 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:25.389325 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:25.389331 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:25.389344 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0925 15:13:25.389642 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:25.389653 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:25.390776 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:25.390783 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:25.392534 (+ 1751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:25.392549 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:25.393245 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:25.393305 (+ 60us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:25.393490 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:25.393534 (+ 44us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 15:13:25.436035 (+ 42501us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 15:13:25.436368 (+ 333us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.12.8.62:38999: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":301,"threads_started":1}
I20260925 15:13:25.438112 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 17777
I20260925 15:13:25.439235 17838 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:13:25.534277 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 17777
2026-09-25T15:13:25Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (707 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (400 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (401 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:13:26 dist-test-slave-p468 krb5kdc[17887](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:13:26 dist-test-slave-p468 krb5kdc[17887](info): set up 2 sockets
Sep 25 15:13:26 dist-test-slave-p468 krb5kdc[17887](info): commencing operation
krb5kdc: starting...
W20260925 15:13:27.454806 12320 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/krb5kdc/test-user.keytab.
Sep 25 15:13:27 dist-test-slave-p468 krb5kdc[17887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349207, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:13:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:13:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.8.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab.
Entry for principal kudu/127.12.8.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab.
I20260925 15:13:27.654431 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.62:34911
--webserver_interface=127.12.8.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab
--principal=kudu/127.12.8.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:35275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.62:34911
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:13:27.969025 17900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:13:27.969370 17900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:13:27.969522 17900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:13:27.976658 17900 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:13:27.976799 17900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:13:27.976902 17900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:13:27.977005 17900 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:13:27.977088 17900 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:13:27.986706 17900 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:35275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.8.62:34911
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab
--principal=kudu/127.12.8.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.62:34911
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.8.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:13:27.987973 17900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:13:27.989261 17900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:13:28.001034 17906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:28.001180 17905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:28.001138 17908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:13:28.002264 17900 server_base.cc:1034] running on GCE node
I20260925 15:13:28.002960 17900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:13:28.004273 17900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:13:28.005502 17900 hybrid_clock.cc:648] HybridClock initialized: now 1790349208005445 us; error 37 us; skew 500 ppm
I20260925 15:13:28.005887 17900 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:13:28 dist-test-slave-p468 krb5kdc[17887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349208, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:13:28.009053 17900 init.cc:377] Logged in from keytab as kudu/127.12.8.62@KRBTEST.COM (short username kudu)
I20260925 15:13:28.010675 17900 webserver.cc:460] Webserver started at http://127.12.8.62:38265/ using document root <none> and password file <none>
I20260925 15:13:28.011351 17900 fs_manager.cc:359] Metadata directory not provided
I20260925 15:13:28.011511 17900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:13:28.011808 17900 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:13:28.014734 17900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/minicluster-data/master-0/data/instance:
uuid: "a54fa911d1ff42b087556ea389b268d1"
format_stamp: "Formatted at 2026-09-25 15:13:28 on dist-test-slave-p468"
server_key: "b0ee5331cf6b3aadcc14af075ba7f96d"
server_key_iv: "f4f88a5cf469b96fc6c49f2767182d9c"
server_key_version: "encryptionkey@0"
I20260925 15:13:28.015513 17900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/minicluster-data/master-0/wal/instance:
uuid: "a54fa911d1ff42b087556ea389b268d1"
format_stamp: "Formatted at 2026-09-25 15:13:28 on dist-test-slave-p468"
server_key: "b0ee5331cf6b3aadcc14af075ba7f96d"
server_key_iv: "f4f88a5cf469b96fc6c49f2767182d9c"
server_key_version: "encryptionkey@0"
I20260925 15:13:28.020581 17900 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260925 15:13:28.024171 17915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:28.025739 17900 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 15:13:28.025929 17900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/minicluster-data/master-0/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/minicluster-data/master-0/wal
uuid: "a54fa911d1ff42b087556ea389b268d1"
format_stamp: "Formatted at 2026-09-25 15:13:28 on dist-test-slave-p468"
server_key: "b0ee5331cf6b3aadcc14af075ba7f96d"
server_key_iv: "f4f88a5cf469b96fc6c49f2767182d9c"
server_key_version: "encryptionkey@0"
I20260925 15:13:28.026165 17900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:13:28.040570 17900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:13:28.044473 17900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:13:28.044759 17900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:13:28.063083 17900 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.62:34911
I20260925 15:13:28.063124 17966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.62:34911 every 8 connection(s)
I20260925 15:13:28.064373 17900 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
I20260925 15:13:28.068222 17967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:13:28.072914 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 17900
I20260925 15:13:28.073143 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/minicluster-data/master-0/wal/instance
I20260925 15:13:28.074044 12320 external_mini_cluster.cc:1428] Setting key 9ac4791be5411087e63e852d718dd347
I20260925 15:13:28.080869 17967 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a54fa911d1ff42b087556ea389b268d1: Bootstrap starting.
I20260925 15:13:28.084661 17967 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a54fa911d1ff42b087556ea389b268d1: Neither blocks nor log segments found. Creating new log.
I20260925 15:13:28.085862 17967 log.cc:824] T 00000000000000000000000000000000 P a54fa911d1ff42b087556ea389b268d1: Log is configured to *not* fsync() on all Append() calls
I20260925 15:13:28.089232 17967 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a54fa911d1ff42b087556ea389b268d1: No bootstrap required, opened a new log
Sep 25 15:13:28 dist-test-slave-p468 krb5kdc[17887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349207, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
I20260925 15:13:28.099270 17970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:28.076130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54120 (local address 127.12.8.62:34911)
0925 15:13:28.076882 (+ 752us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:28.076892 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:28.076930 (+ 38us) server_negotiation.cc:407] Connection header received
0925 15:13:28.079167 (+ 2237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:28.079186 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:28.079549 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:28.079876 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:28.081762 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:28.083160 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:28.085120 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:28.085824 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:28.085958 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:28.090799 (+ 4841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:28.090810 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:28.090832 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:28.090878 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:28.094886 (+ 4008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:28.095526 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:28.095535 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:28.095543 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:28.095763 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:28.096105 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:28.096110 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:28.096114 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:28.096815 (+ 701us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:28.097109 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:28.097414 (+ 305us) server_negotiation.cc:299] Negotiation successful
0925 15:13:28.097855 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":383,"threads_started":1}
I20260925 15:13:28.099974 17967 raft_consensus.cc:348] T 00000000000000000000000000000000 P a54fa911d1ff42b087556ea389b268d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a54fa911d1ff42b087556ea389b268d1" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 34911 } }
I20260925 15:13:28.100329 17967 raft_consensus.cc:374] T 00000000000000000000000000000000 P a54fa911d1ff42b087556ea389b268d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:13:28.100449 17967 raft_consensus.cc:729] T 00000000000000000000000000000000 P a54fa911d1ff42b087556ea389b268d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a54fa911d1ff42b087556ea389b268d1, State: Initialized, Role: FOLLOWER
I20260925 15:13:28.101092 17967 consensus_queue.cc:260] T 00000000000000000000000000000000 P a54fa911d1ff42b087556ea389b268d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a54fa911d1ff42b087556ea389b268d1" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 34911 } }
I20260925 15:13:28.101383 17967 raft_consensus.cc:388] T 00000000000000000000000000000000 P a54fa911d1ff42b087556ea389b268d1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:13:28.101519 17967 raft_consensus.cc:482] T 00000000000000000000000000000000 P a54fa911d1ff42b087556ea389b268d1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:13:28.101697 17967 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a54fa911d1ff42b087556ea389b268d1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:13:28.103868 17967 raft_consensus.cc:504] T 00000000000000000000000000000000 P a54fa911d1ff42b087556ea389b268d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a54fa911d1ff42b087556ea389b268d1" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 34911 } }
I20260925 15:13:28.104286 17967 leader_election.cc:302] T 00000000000000000000000000000000 P a54fa911d1ff42b087556ea389b268d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a54fa911d1ff42b087556ea389b268d1; no voters:
I20260925 15:13:28.104789 17967 leader_election.cc:288] T 00000000000000000000000000000000 P a54fa911d1ff42b087556ea389b268d1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:13:28.104969 17972 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a54fa911d1ff42b087556ea389b268d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:13:28.105581 17972 raft_consensus.cc:686] T 00000000000000000000000000000000 P a54fa911d1ff42b087556ea389b268d1 [term 1 LEADER]: Becoming Leader. State: Replica: a54fa911d1ff42b087556ea389b268d1, State: Running, Role: LEADER
I20260925 15:13:28.106213 17972 consensus_queue.cc:237] T 00000000000000000000000000000000 P a54fa911d1ff42b087556ea389b268d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a54fa911d1ff42b087556ea389b268d1" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 34911 } }
I20260925 15:13:28.107430 17967 sys_catalog.cc:564] T 00000000000000000000000000000000 P a54fa911d1ff42b087556ea389b268d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:13:28.108668 17974 sys_catalog.cc:455] T 00000000000000000000000000000000 P a54fa911d1ff42b087556ea389b268d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a54fa911d1ff42b087556ea389b268d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a54fa911d1ff42b087556ea389b268d1" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 34911 } } }
I20260925 15:13:28.109100 17974 sys_catalog.cc:458] T 00000000000000000000000000000000 P a54fa911d1ff42b087556ea389b268d1 [sys.catalog]: This master's current role is: LEADER
I20260925 15:13:28.109021 17973 sys_catalog.cc:455] T 00000000000000000000000000000000 P a54fa911d1ff42b087556ea389b268d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a54fa911d1ff42b087556ea389b268d1. Latest consensus state: current_term: 1 leader_uuid: "a54fa911d1ff42b087556ea389b268d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a54fa911d1ff42b087556ea389b268d1" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 34911 } } }
I20260925 15:13:28.109310 17973 sys_catalog.cc:458] T 00000000000000000000000000000000 P a54fa911d1ff42b087556ea389b268d1 [sys.catalog]: This master's current role is: LEADER
I20260925 15:13:28.110749 17977 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:13:28.115538 17977 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:13:28.125480 17977 catalog_manager.cc:1347] Generated new cluster ID: 52390440685a46f98992871fbcea424f
I20260925 15:13:28.125592 17977 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:13:28.141855 17977 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:13:28.143285 17977 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:13:28.155714 17977 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a54fa911d1ff42b087556ea389b268d1: Generated new TSK 0
I20260925 15:13:28.156522 17977 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.8.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.1.keytab.
Entry for principal kudu/127.12.8.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.1.keytab.
I20260925 15:13:28.218133 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.1:0
--local_ip_for_outbound_sockets=127.12.8.1
--webserver_interface=127.12.8.1
--webserver_port=0
--tserver_master_addrs=127.12.8.62:34911
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.1.keytab
--principal=kudu/127.12.8.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:35275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:13:28.557965 17993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:13:28.558295 17993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:13:28.558475 17993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:13:28.565305 17993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:13:28.565400 17993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:13:28.565510 17993 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.1
I20260925 15:13:28.574994 17993 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:35275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.1.keytab
--principal=kudu/127.12.8.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.8.1
--webserver_port=0
--tserver_master_addrs=127.12.8.62:34911
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.1
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:13:28.576131 17993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:13:28.577376 17993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:13:28.589900 17998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:28.589929 18001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:28.589860 17999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:13:28.590812 17993 server_base.cc:1034] running on GCE node
I20260925 15:13:28.591383 17993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:13:28.592253 17993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:13:28.593472 17993 hybrid_clock.cc:648] HybridClock initialized: now 1790349208593436 us; error 51 us; skew 500 ppm
I20260925 15:13:28.593783 17993 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:13:28 dist-test-slave-p468 krb5kdc[17887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349208, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:13:28.596841 17993 init.cc:377] Logged in from keytab as kudu/127.12.8.1@KRBTEST.COM (short username kudu)
I20260925 15:13:28.598495 17993 webserver.cc:460] Webserver started at http://127.12.8.1:43015/ using document root <none> and password file <none>
I20260925 15:13:28.599296 17993 fs_manager.cc:359] Metadata directory not provided
I20260925 15:13:28.599393 17993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:13:28.599629 17993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:13:28.602416 17993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/minicluster-data/ts-0/data/instance:
uuid: "57b14d2c633a426b9bb332942d70ae3e"
format_stamp: "Formatted at 2026-09-25 15:13:28 on dist-test-slave-p468"
server_key: "299ceac0a4d65cdca8d3d31d4f741241"
server_key_iv: "2b5e8a3bc0bd6727f3213657011995c4"
server_key_version: "encryptionkey@0"
I20260925 15:13:28.603174 17993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/minicluster-data/ts-0/wal/instance:
uuid: "57b14d2c633a426b9bb332942d70ae3e"
format_stamp: "Formatted at 2026-09-25 15:13:28 on dist-test-slave-p468"
server_key: "299ceac0a4d65cdca8d3d31d4f741241"
server_key_iv: "2b5e8a3bc0bd6727f3213657011995c4"
server_key_version: "encryptionkey@0"
I20260925 15:13:28.607931 17993 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 15:13:28.611187 18008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:28.612376 17993 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 15:13:28.612522 17993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/minicluster-data/ts-0/wal
uuid: "57b14d2c633a426b9bb332942d70ae3e"
format_stamp: "Formatted at 2026-09-25 15:13:28 on dist-test-slave-p468"
server_key: "299ceac0a4d65cdca8d3d31d4f741241"
server_key_iv: "2b5e8a3bc0bd6727f3213657011995c4"
server_key_version: "encryptionkey@0"
I20260925 15:13:28.612711 17993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:13:28.627861 17993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:13:28.631742 17993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:13:28.632006 17993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:13:28.632912 17993 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:13:28.634142 17993 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:13:28.634277 17993 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:28.634385 17993 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:13:28.634433 17993 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:28.668129 17993 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.1:35693
I20260925 15:13:28.668184 18120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.1:35693 every 8 connection(s)
I20260925 15:13:28.669402 17993 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/minicluster-data/ts-0/data/info.pb
I20260925 15:13:28.670571 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 17993
I20260925 15:13:28.670759 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/minicluster-data/ts-0/wal/instance
I20260925 15:13:28.671614 12320 external_mini_cluster.cc:1428] Setting key 03b6c0ea8efc76f682f9f937655e386b
Sep 25 15:13:28 dist-test-slave-p468 krb5kdc[17887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349208, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.1@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
I20260925 15:13:28.691306 18123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:28.672209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.1:41391 (local address 127.12.8.62:34911)
0925 15:13:28.672643 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:28.672648 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:28.673834 (+ 1186us) server_negotiation.cc:407] Connection header received
0925 15:13:28.675708 (+ 1874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:28.675716 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:28.675801 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:28.675921 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:28.677331 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:28.678512 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:28.680033 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:28.680434 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:28.680573 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:28.684233 (+ 3660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:28.684242 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:28.684250 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:28.684295 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:28.687963 (+ 3668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:28.688653 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:28.688663 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:28.688669 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:28.688830 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:28.689250 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:28.689257 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:28.689264 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:28.689666 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:28.689824 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:28.690653 (+ 829us) server_negotiation.cc:299] Negotiation successful
0925 15:13:28.690902 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":215,"threads_started":1}
WARNING: no policy specified for kudu/127.12.8.2@KRBTEST.COM; defaulting to no policy
I20260925 15:13:28.692028 18124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:28.672840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:34911 (local address 127.12.8.1:41391)
0925 15:13:28.673673 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:28.673727 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:28.675453 (+ 1726us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:28.676109 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:28.676117 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:28.676600 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:13:28.677161 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:28.677178 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:28.678660 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:28.678664 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:28.679881 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:28.679893 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:28.680673 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:28.680681 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:28.680928 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:28.681597 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:13:28.681619 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:13:28.684007 (+ 2388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:13:28.688098 (+ 4091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:13:28.688105 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:13:28.688118 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:13:28.688523 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:13:28.688947 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:13:28.688950 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:13:28.688952 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:13:28.689125 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:13:28.689824 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:13:28.689830 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:13:28.690256 (+ 426us) client_negotiation.cc:769] Sending connection context
0925 15:13:28.690514 (+ 258us) client_negotiation.cc:240] Negotiation successful
0925 15:13:28.690905 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":370,"threads_started":1}
I20260925 15:13:28.693684 18121 heartbeater.cc:347] Connected to a master server at 127.12.8.62:34911
I20260925 15:13:28.694024 18121 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.2@KRBTEST.COM" created.
I20260925 15:13:28.694911 18121 heartbeater.cc:511] Master 127.12.8.62:34911 requested a full tablet report, sending...
I20260925 15:13:28.697212 17932 ts_manager.cc:194] Registered new tserver with Master: 57b14d2c633a426b9bb332942d70ae3e (127.12.8.1:35693)
I20260925 15:13:28.698705 17932 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.1@KRBTEST.COM'} at 127.12.8.1:41391
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.2.keytab.
Entry for principal kudu/127.12.8.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.2.keytab.
I20260925 15:13:28.721611 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.2:0
--local_ip_for_outbound_sockets=127.12.8.2
--webserver_interface=127.12.8.2
--webserver_port=0
--tserver_master_addrs=127.12.8.62:34911
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.2.keytab
--principal=kudu/127.12.8.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:35275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:13:29.039692 18127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:13:29.039958 18127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:13:29.040119 18127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:13:29.047261 18127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:13:29.047362 18127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:13:29.047564 18127 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.2
I20260925 15:13:29.057060 18127 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:35275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.2.keytab
--principal=kudu/127.12.8.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.8.2
--webserver_port=0
--tserver_master_addrs=127.12.8.62:34911
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.2
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:13:29.058365 18127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:13:29.059739 18127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:13:29.074470 18133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:29.074819 18135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:29.074939 18132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:13:29.075835 18127 server_base.cc:1034] running on GCE node
I20260925 15:13:29.076538 18127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:13:29.077636 18127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:13:29.078903 18127 hybrid_clock.cc:648] HybridClock initialized: now 1790349209078853 us; error 62 us; skew 500 ppm
I20260925 15:13:29.079285 18127 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:13:29 dist-test-slave-p468 krb5kdc[17887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349209, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:13:29.083283 18127 init.cc:377] Logged in from keytab as kudu/127.12.8.2@KRBTEST.COM (short username kudu)
I20260925 15:13:29.084777 18127 webserver.cc:460] Webserver started at http://127.12.8.2:37537/ using document root <none> and password file <none>
I20260925 15:13:29.085445 18127 fs_manager.cc:359] Metadata directory not provided
I20260925 15:13:29.085587 18127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:13:29.085888 18127 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:13:29.088729 18127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/minicluster-data/ts-1/data/instance:
uuid: "aa487bc36f2f4add9c7714430d9bfa74"
format_stamp: "Formatted at 2026-09-25 15:13:29 on dist-test-slave-p468"
server_key: "259f1e43ec997fc14ef51813581fdd98"
server_key_iv: "aa48f10f35878707070d877d3a901823"
server_key_version: "encryptionkey@0"
I20260925 15:13:29.089553 18127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/minicluster-data/ts-1/wal/instance:
uuid: "aa487bc36f2f4add9c7714430d9bfa74"
format_stamp: "Formatted at 2026-09-25 15:13:29 on dist-test-slave-p468"
server_key: "259f1e43ec997fc14ef51813581fdd98"
server_key_iv: "aa48f10f35878707070d877d3a901823"
server_key_version: "encryptionkey@0"
I20260925 15:13:29.094462 18127 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 15:13:29.097755 18142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:29.099005 18127 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:13:29.099189 18127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/minicluster-data/ts-1/wal
uuid: "aa487bc36f2f4add9c7714430d9bfa74"
format_stamp: "Formatted at 2026-09-25 15:13:29 on dist-test-slave-p468"
server_key: "259f1e43ec997fc14ef51813581fdd98"
server_key_iv: "aa48f10f35878707070d877d3a901823"
server_key_version: "encryptionkey@0"
I20260925 15:13:29.099381 18127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:13:29.134289 18127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:13:29.138583 18127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:13:29.138881 18127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:13:29.139811 18127 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:13:29.141165 18127 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:13:29.141273 18127 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:29.141422 18127 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:13:29.141495 18127 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:29.175624 18127 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.2:41605
I20260925 15:13:29.175710 18254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.2:41605 every 8 connection(s)
I20260925 15:13:29.176842 18127 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/minicluster-data/ts-1/data/info.pb
I20260925 15:13:29.181041 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 18127
I20260925 15:13:29.181222 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/minicluster-data/ts-1/wal/instance
I20260925 15:13:29.181648 12320 external_mini_cluster.cc:1428] Setting key 0fb53469c6b355eb64df32397235f7b2
Sep 25 15:13:29 dist-test-slave-p468 krb5kdc[17887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349209, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.2@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
I20260925 15:13:29.197425 18123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:29.179449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.2:38273 (local address 127.12.8.62:34911)
0925 15:13:29.179645 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:29.179649 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:29.180831 (+ 1182us) server_negotiation.cc:407] Connection header received
0925 15:13:29.183055 (+ 2224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:29.183061 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:29.183155 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:29.183297 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:29.184795 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:29.186115 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:29.187638 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:29.188510 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:29.188664 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:29.191885 (+ 3221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:29.191890 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:29.191893 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:29.191921 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:29.194585 (+ 2664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:29.195269 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:29.195276 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:29.195279 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:29.195403 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:29.195808 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:29.195816 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:29.195819 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:29.196066 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:29.196228 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:29.196828 (+ 600us) server_negotiation.cc:299] Negotiation successful
0925 15:13:29.197042 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 15:13:29.198354 18257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:29.179941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:34911 (local address 127.12.8.2:38273)
0925 15:13:29.180664 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:29.180711 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:29.182785 (+ 2074us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:29.183581 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:29.183595 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:29.184094 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:13:29.184612 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:29.184628 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:29.186296 (+ 1668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:29.186300 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:29.187486 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:29.187497 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:29.188708 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:29.188713 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:29.188912 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:29.189599 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:13:29.189619 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:13:29.191680 (+ 2061us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:13:29.194719 (+ 3039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:13:29.194728 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:13:29.194743 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:13:29.195153 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:13:29.195518 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:13:29.195523 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:13:29.195527 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:13:29.195701 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:13:29.196195 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:13:29.196203 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:13:29.196500 (+ 297us) client_negotiation.cc:769] Sending connection context
0925 15:13:29.196758 (+ 258us) client_negotiation.cc:240] Negotiation successful
0925 15:13:29.197235 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":308,"threads_started":1}
I20260925 15:13:29.199739 18255 heartbeater.cc:347] Connected to a master server at 127.12.8.62:34911
I20260925 15:13:29.200027 18255 heartbeater.cc:464] Registering TS with master...
I20260925 15:13:29.200676 18255 heartbeater.cc:511] Master 127.12.8.62:34911 requested a full tablet report, sending...
I20260925 15:13:29.202175 17932 ts_manager.cc:194] Registered new tserver with Master: aa487bc36f2f4add9c7714430d9bfa74 (127.12.8.2:41605)
WARNING: no policy specified for kudu/127.12.8.3@KRBTEST.COM; defaulting to no policy
I20260925 15:13:29.203276 17932 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.2@KRBTEST.COM'} at 127.12.8.2:38273
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.3.keytab.
Entry for principal kudu/127.12.8.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.3.keytab.
I20260925 15:13:29.230177 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.3:0
--local_ip_for_outbound_sockets=127.12.8.3
--webserver_interface=127.12.8.3
--webserver_port=0
--tserver_master_addrs=127.12.8.62:34911
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.3.keytab
--principal=kudu/127.12.8.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:35275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:13:29.542709 18260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:13:29.542954 18260 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:13:29.543083 18260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:13:29.550037 18260 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:13:29.550137 18260 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:13:29.550343 18260 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.3
I20260925 15:13:29.559600 18260 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:35275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.3.keytab
--principal=kudu/127.12.8.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.8.3
--webserver_port=0
--tserver_master_addrs=127.12.8.62:34911
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.3
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:13:29.560772 18260 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:13:29.562002 18260 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:13:29.574714 18265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:29.574936 18266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:29.574908 18268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:13:29.575867 18260 server_base.cc:1034] running on GCE node
I20260925 15:13:29.576463 18260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:13:29.577414 18260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:13:29.578634 18260 hybrid_clock.cc:648] HybridClock initialized: now 1790349209578580 us; error 45 us; skew 500 ppm
I20260925 15:13:29.578938 18260 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:13:29 dist-test-slave-p468 krb5kdc[17887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349209, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:13:29.581839 18260 init.cc:377] Logged in from keytab as kudu/127.12.8.3@KRBTEST.COM (short username kudu)
I20260925 15:13:29.583408 18260 webserver.cc:460] Webserver started at http://127.12.8.3:39795/ using document root <none> and password file <none>
I20260925 15:13:29.584039 18260 fs_manager.cc:359] Metadata directory not provided
I20260925 15:13:29.584173 18260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:13:29.584468 18260 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:13:29.587298 18260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/minicluster-data/ts-2/data/instance:
uuid: "427359e9b0804ddca57b4f893ff29ed4"
format_stamp: "Formatted at 2026-09-25 15:13:29 on dist-test-slave-p468"
server_key: "5a2a24ba9519ae2a2592fd0d535c6e85"
server_key_iv: "f0047e9cc31d2703edcc6260285c37de"
server_key_version: "encryptionkey@0"
I20260925 15:13:29.588040 18260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/minicluster-data/ts-2/wal/instance:
uuid: "427359e9b0804ddca57b4f893ff29ed4"
format_stamp: "Formatted at 2026-09-25 15:13:29 on dist-test-slave-p468"
server_key: "5a2a24ba9519ae2a2592fd0d535c6e85"
server_key_iv: "f0047e9cc31d2703edcc6260285c37de"
server_key_version: "encryptionkey@0"
I20260925 15:13:29.592814 18260 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 15:13:29.596099 18275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:29.597308 18260 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 15:13:29.597476 18260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/minicluster-data/ts-2/wal
uuid: "427359e9b0804ddca57b4f893ff29ed4"
format_stamp: "Formatted at 2026-09-25 15:13:29 on dist-test-slave-p468"
server_key: "5a2a24ba9519ae2a2592fd0d535c6e85"
server_key_iv: "f0047e9cc31d2703edcc6260285c37de"
server_key_version: "encryptionkey@0"
I20260925 15:13:29.597667 18260 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:13:29.612460 18260 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:13:29.616935 18260 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:13:29.617215 18260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:13:29.618114 18260 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:13:29.619400 18260 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:13:29.619501 18260 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:29.619650 18260 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:13:29.619729 18260 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:29.654721 18260 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.3:40443
I20260925 15:13:29.654765 18387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.3:40443 every 8 connection(s)
I20260925 15:13:29.656077 18260 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/minicluster-data/ts-2/data/info.pb
I20260925 15:13:29.658706 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 18260
I20260925 15:13:29.658896 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/minicluster-data/ts-2/wal/instance
I20260925 15:13:29.659368 12320 external_mini_cluster.cc:1428] Setting key 70000e90bf3384000fb8d727797644af
Sep 25 15:13:29 dist-test-slave-p468 krb5kdc[17887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349209, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.3@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
I20260925 15:13:29.676664 18123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:29.658927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.3:36867 (local address 127.12.8.62:34911)
0925 15:13:29.659167 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:29.659174 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:29.660511 (+ 1337us) server_negotiation.cc:407] Connection header received
0925 15:13:29.662518 (+ 2007us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:29.662522 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:29.662587 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:29.662673 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:29.664171 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:29.664905 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:29.666416 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:29.666792 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:29.666886 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:29.670570 (+ 3684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:29.670575 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:29.670578 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:29.670608 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:29.673317 (+ 2709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:29.674552 (+ 1235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:29.674558 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:29.674561 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:29.674734 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:29.675214 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:29.675218 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:29.675219 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:29.675460 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:29.675562 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:29.676272 (+ 710us) server_negotiation.cc:299] Negotiation successful
0925 15:13:29.676420 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260925 15:13:29.677678 18390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:29.659549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:34911 (local address 127.12.8.3:36867)
0925 15:13:29.660353 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:29.660400 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:29.662283 (+ 1883us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:29.662977 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:29.662985 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:29.663478 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:13:29.664021 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:29.664037 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:29.665026 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:29.665029 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:29.666298 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:29.666309 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:29.666971 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:29.666975 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:29.667163 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:29.668008 (+ 845us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:13:29.668029 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:13:29.670376 (+ 2347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:13:29.673503 (+ 3127us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:13:29.673512 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:13:29.673528 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:13:29.674375 (+ 847us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:13:29.674844 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:13:29.674848 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:13:29.674850 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:13:29.675109 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:13:29.675578 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:13:29.675584 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:13:29.675899 (+ 315us) client_negotiation.cc:769] Sending connection context
0925 15:13:29.676171 (+ 272us) client_negotiation.cc:240] Negotiation successful
0925 15:13:29.676578 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":350,"threads_started":1}
I20260925 15:13:29.678970 18388 heartbeater.cc:347] Connected to a master server at 127.12.8.62:34911
I20260925 15:13:29.679245 18388 heartbeater.cc:464] Registering TS with master...
I20260925 15:13:29.679932 18388 heartbeater.cc:511] Master 127.12.8.62:34911 requested a full tablet report, sending...
I20260925 15:13:29.681203 17932 ts_manager.cc:194] Registered new tserver with Master: 427359e9b0804ddca57b4f893ff29ed4 (127.12.8.3:40443)
I20260925 15:13:29.681969 17932 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.3@KRBTEST.COM'} at 127.12.8.3:36867
I20260925 15:13:29.686825 12320 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:13:29.702426 18121 heartbeater.cc:503] Master 127.12.8.62:34911 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/krb5kdc/test-admin.keytab.
Sep 25 15:13:29 dist-test-slave-p468 krb5kdc[17887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349209, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:13:29.714808 12320 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260925 15:13:29.719794 18395 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:13:29.717451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46426 (local address 127.12.8.1:35693)
0925 15:13:29.718083 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:29.718089 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:29.718114 (+ 25us) server_negotiation.cc:407] Connection header received
0925 15:13:29.718180 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:29.718228 (+ 48us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:29.718463 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:29.718590 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:29.719369 (+ 779us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46426: BlockingRecv error: recv got EOF from 127.0.0.1:46426 (error 108)
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":348,"threads_started":1}
W20260925 15:13:29.719916 12320 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/krb5kdc/krb5cc' of type FILE
Sep 25 15:13:29 dist-test-slave-p468 krb5kdc[17887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349209, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:13:29.721403 12320 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 15:13:29 dist-test-slave-p468 krb5kdc[17887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349209, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:13:29.722832 12320 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 15:13:29 dist-test-slave-p468 krb5kdc[17887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349209, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.1@KRBTEST.COM
I20260925 15:13:29.738135 18395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:29.724297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46438 (local address 127.12.8.1:35693)
0925 15:13:29.724492 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:29.724497 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:29.724733 (+ 236us) server_negotiation.cc:407] Connection header received
0925 15:13:29.724846 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:29.724850 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:29.724904 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:29.724991 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:29.725924 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:29.726965 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:29.728166 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:29.728720 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:29.728898 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:29.732724 (+ 3826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:29.732734 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:29.732748 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:29.732778 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:29.735879 (+ 3101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:29.736438 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:29.736444 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:29.736446 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:29.736550 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:29.736851 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:29.736854 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:29.736856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:29.737171 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:29.737370 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:29.737617 (+ 247us) server_negotiation.cc:299] Negotiation successful
0925 15:13:29.737864 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260925 15:13:29.743592 18395 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:13:29.741886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46452 (local address 127.12.8.1:35693)
0925 15:13:29.742135 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:29.742142 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:29.742378 (+ 236us) server_negotiation.cc:407] Connection header received
0925 15:13:29.742563 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:29.742568 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:29.742646 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:29.742745 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:29.743377 (+ 632us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46452: BlockingRecv error: recv got EOF from 127.0.0.1:46452 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260925 15:13:29.744020 12320 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/krb5kdc/krb5cc)
Sep 25 15:13:29 dist-test-slave-p468 krb5kdc[17887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349209, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:13:29.745391 12320 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 15:13:29 dist-test-slave-p468 krb5kdc[17887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349209, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:13:29.746672 12320 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 15:13:29 dist-test-slave-p468 krb5kdc[17887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349209, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.1@KRBTEST.COM
I20260925 15:13:29.759847 18395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:29.747886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46464 (local address 127.12.8.1:35693)
0925 15:13:29.748068 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:29.748073 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:29.748317 (+ 244us) server_negotiation.cc:407] Connection header received
0925 15:13:29.748486 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:29.748491 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:29.748554 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:29.748650 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:29.749527 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:29.750259 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:29.751421 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:29.751849 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:29.751977 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:29.755203 (+ 3226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:29.755207 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:29.755210 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:29.755246 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:29.757445 (+ 2199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:29.758075 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:29.758088 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:29.758092 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:29.758251 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:29.758614 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:29.758621 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:29.758624 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:29.758871 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:29.759147 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:29.759398 (+ 251us) server_negotiation.cc:299] Negotiation successful
0925 15:13:29.759578 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260925 15:13:29.763705 18395 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:13:29.762290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46470 (local address 127.12.8.1:35693)
0925 15:13:29.762463 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:29.762468 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:29.762659 (+ 191us) server_negotiation.cc:407] Connection header received
0925 15:13:29.762765 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:29.762768 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:29.762827 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:29.762898 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:29.763496 (+ 598us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46470: BlockingRecv error: recv got EOF from 127.0.0.1:46470 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260925 15:13:29.763815 12320 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349162370577-12320-0/krb5kdc/krb5cc)
Sep 25 15:13:29 dist-test-slave-p468 krb5kdc[17887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349209, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:13:29.765154 12320 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 15:13:29 dist-test-slave-p468 krb5kdc[17887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349209, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:13:29.766400 12320 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 15:13:29 dist-test-slave-p468 krb5kdc[17887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349209, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.1@KRBTEST.COM
I20260925 15:13:29.779369 18395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:29.767529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46484 (local address 127.12.8.1:35693)
0925 15:13:29.767731 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:29.767737 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:29.767965 (+ 228us) server_negotiation.cc:407] Connection header received
0925 15:13:29.768152 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:29.768161 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:29.768208 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:29.768303 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:29.769210 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:29.769888 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:29.771216 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:29.771631 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:29.771754 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:29.775132 (+ 3378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:29.775137 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:29.775140 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:29.775167 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:29.777366 (+ 2199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:29.777916 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:29.777920 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:29.777922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:29.778034 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:29.778369 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:29.778372 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:29.778374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:29.778610 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:29.778819 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:29.778995 (+ 176us) server_negotiation.cc:299] Negotiation successful
0925 15:13:29.779134 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260925 15:13:29.780876 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 17993
I20260925 15:13:29.792407 18395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:29.781695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46488 (local address 127.12.8.1:35693)
0925 15:13:29.781868 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:29.781873 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:29.782055 (+ 182us) server_negotiation.cc:407] Connection header received
0925 15:13:29.782305 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:29.782317 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:29.782366 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:29.782459 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:29.783316 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:29.784015 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:29.785356 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:29.785763 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:29.785886 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:29.787860 (+ 1974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:29.787870 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:29.787873 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:29.787899 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:29.790065 (+ 2166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:29.790681 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:29.790688 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:29.790691 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:29.790819 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:29.791178 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:29.791182 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:29.791186 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:29.791407 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:29.791673 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:29.791947 (+ 274us) server_negotiation.cc:299] Negotiation successful
0925 15:13:29.792128 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 15:13:29.793036 18116 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:13:29.877815 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 17993
I20260925 15:13:29.893968 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 18127
Sep 25 15:13:29 dist-test-slave-p468 krb5kdc[17887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349209, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.2@KRBTEST.COM
I20260925 15:13:29.909312 18408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:29.894875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56432 (local address 127.12.8.2:41605)
0925 15:13:29.895370 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:29.895377 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:29.895400 (+ 23us) server_negotiation.cc:407] Connection header received
0925 15:13:29.895458 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:29.895464 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:29.895669 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:29.895789 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:29.896731 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:29.897685 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:29.898925 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:29.899432 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:29.899531 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:29.903738 (+ 4207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:29.903748 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:29.903761 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:29.903797 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:29.906581 (+ 2784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:29.907244 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:29.907251 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:29.907254 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:29.907362 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:29.907768 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:29.907775 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:29.907778 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:29.908137 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:29.908428 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:29.908751 (+ 323us) server_negotiation.cc:299] Negotiation successful
0925 15:13:29.908989 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":245,"threads_started":1}
I20260925 15:13:29.910605 18250 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:13:29.993969 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 18127
I20260925 15:13:30.009747 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 18260
Sep 25 15:13:30 dist-test-slave-p468 krb5kdc[17887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349209, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.3@KRBTEST.COM
I20260925 15:13:30.026310 18410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:30.011072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47714 (local address 127.12.8.3:40443)
0925 15:13:30.011738 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:30.011745 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:30.011776 (+ 31us) server_negotiation.cc:407] Connection header received
0925 15:13:30.011838 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:30.011845 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:30.012079 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:30.012234 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:30.013207 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:30.014417 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:30.015690 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:30.016386 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:30.016488 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:30.020480 (+ 3992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:30.020493 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:30.020512 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:30.020548 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:30.023595 (+ 3047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:30.024298 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:30.024306 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:30.024308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:30.024425 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:30.024786 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:30.024792 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:30.024795 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:30.025123 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:30.025462 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:30.025700 (+ 238us) server_negotiation.cc:299] Negotiation successful
0925 15:13:30.025947 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":355,"threads_started":1}
I20260925 15:13:30.027559 18383 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:13:30.110222 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 18260
I20260925 15:13:30.125793 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 17900
I20260925 15:13:30.127094 17962 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:13:30.231840 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 17900
2026-09-25T15:13:30Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3901 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:13:30 dist-test-slave-p468 krb5kdc[18415](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:13:30 dist-test-slave-p468 krb5kdc[18415](info): set up 2 sockets
Sep 25 15:13:30 dist-test-slave-p468 krb5kdc[18415](info): commencing operation
krb5kdc: starting...
W20260925 15:13:31.315404 12320 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.001s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/krb5kdc/test-user.keytab.
Sep 25 15:13:31 dist-test-slave-p468 krb5kdc[18415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349211, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:13:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:13:31Z Disabled control of system clock
WARNING: no policy specified for oryx/127.12.8.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.8.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.8.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/krb5kdc/oryx_127.12.8.62.keytab.
Entry for principal oryx/127.12.8.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/krb5kdc/oryx_127.12.8.62.keytab.
I20260925 15:13:31.521811 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.62:44383
--webserver_interface=127.12.8.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/krb5kdc/oryx_127.12.8.62.keytab
--principal=oryx/127.12.8.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:36421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.62:44383
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:13:31.833505 18428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:13:31.833827 18428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:13:31.834007 18428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:13:31.841156 18428 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:13:31.841305 18428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:13:31.841418 18428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:13:31.841526 18428 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:13:31.841591 18428 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:13:31.851364 18428 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:36421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.8.62:44383
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/krb5kdc/oryx_127.12.8.62.keytab
--principal=oryx/127.12.8.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.62:44383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.8.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:13:31.852628 18428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:13:31.853945 18428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:13:31.868103 18434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:31.868554 18433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:13:31.869387 18428 server_base.cc:1034] running on GCE node
W20260925 15:13:31.869498 18436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:13:31.870354 18428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:13:31.871655 18428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:13:31.872892 18428 hybrid_clock.cc:648] HybridClock initialized: now 1790349211872860 us; error 44 us; skew 500 ppm
I20260925 15:13:31.873310 18428 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:13:31 dist-test-slave-p468 krb5kdc[18415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349211, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.8.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:13:31.879554 18428 init.cc:377] Logged in from keytab as oryx/127.12.8.62@KRBTEST.COM (short username oryx)
I20260925 15:13:31.881028 18428 webserver.cc:460] Webserver started at http://127.12.8.62:32877/ using document root <none> and password file <none>
I20260925 15:13:31.881697 18428 fs_manager.cc:359] Metadata directory not provided
I20260925 15:13:31.881829 18428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:13:31.882135 18428 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:13:31.885171 18428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/master-0/data/instance:
uuid: "c1233ca1ad3745eca6025d08fc59d7f3"
format_stamp: "Formatted at 2026-09-25 15:13:31 on dist-test-slave-p468"
server_key: "452bed7f38c3f5d30ba23dc02cdfc171"
server_key_iv: "b9e086646e26b7bcbf6e0a55dac57cba"
server_key_version: "encryptionkey@0"
I20260925 15:13:31.885916 18428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/master-0/wal/instance:
uuid: "c1233ca1ad3745eca6025d08fc59d7f3"
format_stamp: "Formatted at 2026-09-25 15:13:31 on dist-test-slave-p468"
server_key: "452bed7f38c3f5d30ba23dc02cdfc171"
server_key_iv: "b9e086646e26b7bcbf6e0a55dac57cba"
server_key_version: "encryptionkey@0"
I20260925 15:13:31.890864 18428 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260925 15:13:31.894003 18443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:31.895285 18428 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 15:13:31.895412 18428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/master-0/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/master-0/wal
uuid: "c1233ca1ad3745eca6025d08fc59d7f3"
format_stamp: "Formatted at 2026-09-25 15:13:31 on dist-test-slave-p468"
server_key: "452bed7f38c3f5d30ba23dc02cdfc171"
server_key_iv: "b9e086646e26b7bcbf6e0a55dac57cba"
server_key_version: "encryptionkey@0"
I20260925 15:13:31.895557 18428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:13:31.916178 18428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:13:31.919845 18428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:13:31.920076 18428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:13:31.938079 18428 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.62:44383
I20260925 15:13:31.938110 18494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.62:44383 every 8 connection(s)
I20260925 15:13:31.939428 18428 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
I20260925 15:13:31.942855 18495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:13:31.949649 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 18428
I20260925 15:13:31.949858 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/master-0/wal/instance
I20260925 15:13:31.950587 12320 external_mini_cluster.cc:1428] Setting key 6f01c75512e9dff9218817ea06f5eb5b
I20260925 15:13:31.952682 18495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1233ca1ad3745eca6025d08fc59d7f3: Bootstrap starting.
I20260925 15:13:31.956737 18495 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c1233ca1ad3745eca6025d08fc59d7f3: Neither blocks nor log segments found. Creating new log.
I20260925 15:13:31.957827 18495 log.cc:824] T 00000000000000000000000000000000 P c1233ca1ad3745eca6025d08fc59d7f3: Log is configured to *not* fsync() on all Append() calls
I20260925 15:13:31.960819 18495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1233ca1ad3745eca6025d08fc59d7f3: No bootstrap required, opened a new log
Sep 25 15:13:31 dist-test-slave-p468 krb5kdc[18415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349211, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM
I20260925 15:13:31.971225 18495 raft_consensus.cc:348] T 00000000000000000000000000000000 P c1233ca1ad3745eca6025d08fc59d7f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1233ca1ad3745eca6025d08fc59d7f3" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 44383 } }
I20260925 15:13:31.971652 18495 raft_consensus.cc:374] T 00000000000000000000000000000000 P c1233ca1ad3745eca6025d08fc59d7f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:13:31.971747 18495 raft_consensus.cc:729] T 00000000000000000000000000000000 P c1233ca1ad3745eca6025d08fc59d7f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1233ca1ad3745eca6025d08fc59d7f3, State: Initialized, Role: FOLLOWER
I20260925 15:13:31.972409 18495 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1233ca1ad3745eca6025d08fc59d7f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1233ca1ad3745eca6025d08fc59d7f3" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 44383 } }
I20260925 15:13:31.972659 18495 raft_consensus.cc:388] T 00000000000000000000000000000000 P c1233ca1ad3745eca6025d08fc59d7f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:13:31.972792 18495 raft_consensus.cc:482] T 00000000000000000000000000000000 P c1233ca1ad3745eca6025d08fc59d7f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:13:31.972990 18495 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c1233ca1ad3745eca6025d08fc59d7f3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:13:31.975416 18495 raft_consensus.cc:504] T 00000000000000000000000000000000 P c1233ca1ad3745eca6025d08fc59d7f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1233ca1ad3745eca6025d08fc59d7f3" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 44383 } }
I20260925 15:13:31.975490 18498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:31.952643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46924 (local address 127.12.8.62:44383)
0925 15:13:31.953439 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:31.953454 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:31.953500 (+ 46us) server_negotiation.cc:407] Connection header received
0925 15:13:31.955947 (+ 2447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:31.955973 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:31.956416 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:31.956848 (+ 432us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:31.958250 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:31.960112 (+ 1862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:31.961409 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:31.962453 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:31.962643 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:31.967401 (+ 4758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:31.967411 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:31.967430 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:31.967482 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:31.971330 (+ 3848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:31.972102 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:31.972109 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:31.972118 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:31.972328 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:31.972811 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:31.972815 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:31.972817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:31.973417 (+ 600us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:31.973764 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:31.974139 (+ 375us) server_negotiation.cc:299] Negotiation successful
0925 15:13:31.974545 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":362,"threads_started":1}
I20260925 15:13:31.975991 18495 leader_election.cc:302] T 00000000000000000000000000000000 P c1233ca1ad3745eca6025d08fc59d7f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c1233ca1ad3745eca6025d08fc59d7f3; no voters:
I20260925 15:13:31.976637 18495 leader_election.cc:288] T 00000000000000000000000000000000 P c1233ca1ad3745eca6025d08fc59d7f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:13:31.976799 18500 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c1233ca1ad3745eca6025d08fc59d7f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:13:31.978629 18500 raft_consensus.cc:686] T 00000000000000000000000000000000 P c1233ca1ad3745eca6025d08fc59d7f3 [term 1 LEADER]: Becoming Leader. State: Replica: c1233ca1ad3745eca6025d08fc59d7f3, State: Running, Role: LEADER
I20260925 15:13:31.978871 18495 sys_catalog.cc:564] T 00000000000000000000000000000000 P c1233ca1ad3745eca6025d08fc59d7f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:13:31.979203 18500 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1233ca1ad3745eca6025d08fc59d7f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1233ca1ad3745eca6025d08fc59d7f3" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 44383 } }
I20260925 15:13:31.981684 18502 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1233ca1ad3745eca6025d08fc59d7f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1233ca1ad3745eca6025d08fc59d7f3. Latest consensus state: current_term: 1 leader_uuid: "c1233ca1ad3745eca6025d08fc59d7f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1233ca1ad3745eca6025d08fc59d7f3" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 44383 } } }
I20260925 15:13:31.982008 18502 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1233ca1ad3745eca6025d08fc59d7f3 [sys.catalog]: This master's current role is: LEADER
I20260925 15:13:31.982537 18501 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1233ca1ad3745eca6025d08fc59d7f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c1233ca1ad3745eca6025d08fc59d7f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1233ca1ad3745eca6025d08fc59d7f3" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 44383 } } }
I20260925 15:13:31.982914 18501 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1233ca1ad3745eca6025d08fc59d7f3 [sys.catalog]: This master's current role is: LEADER
W20260925 15:13:31.991773 18517 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c1233ca1ad3745eca6025d08fc59d7f3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:13:31.991902 18517 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:13:31.992475 18509 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:13:31.994014 18509 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:13:32.001942 18509 catalog_manager.cc:1347] Generated new cluster ID: 5911822a4ce34211bb32228d45a52d8f
I20260925 15:13:32.002049 18509 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:13:32.022162 18509 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:13:32.024049 18509 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:13:32.035565 18509 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c1233ca1ad3745eca6025d08fc59d7f3: Generated new TSK 0
I20260925 15:13:32.036407 18509 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.12.8.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.8.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.8.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/krb5kdc/oryx_127.12.8.1.keytab.
Entry for principal oryx/127.12.8.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/krb5kdc/oryx_127.12.8.1.keytab.
I20260925 15:13:32.089437 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.1:0
--local_ip_for_outbound_sockets=127.12.8.1
--webserver_interface=127.12.8.1
--webserver_port=0
--tserver_master_addrs=127.12.8.62:44383
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/krb5kdc/oryx_127.12.8.1.keytab
--principal=oryx/127.12.8.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:36421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:13:32.408890 18522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:13:32.409202 18522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:13:32.409374 18522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:13:32.416647 18522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:13:32.416806 18522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:13:32.416985 18522 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.1
I20260925 15:13:32.426535 18522 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:36421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/krb5kdc/oryx_127.12.8.1.keytab
--principal=oryx/127.12.8.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.8.1
--webserver_port=0
--tserver_master_addrs=127.12.8.62:44383
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.1
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:13:32.427755 18522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:13:32.429077 18522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:13:32.441606 18530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:32.441649 18528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:32.441649 18527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:13:32.442405 18522 server_base.cc:1034] running on GCE node
I20260925 15:13:32.442950 18522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:13:32.443848 18522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:13:32.445062 18522 hybrid_clock.cc:648] HybridClock initialized: now 1790349212445021 us; error 60 us; skew 500 ppm
I20260925 15:13:32.445353 18522 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:13:32 dist-test-slave-p468 krb5kdc[18415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349212, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.8.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:13:32.448289 18522 init.cc:377] Logged in from keytab as oryx/127.12.8.1@KRBTEST.COM (short username oryx)
I20260925 15:13:32.449560 18522 webserver.cc:460] Webserver started at http://127.12.8.1:37297/ using document root <none> and password file <none>
I20260925 15:13:32.450229 18522 fs_manager.cc:359] Metadata directory not provided
I20260925 15:13:32.450379 18522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:13:32.450670 18522 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:13:32.453385 18522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/ts-0/data/instance:
uuid: "372093292e564acaae5d8d6370b2a26c"
format_stamp: "Formatted at 2026-09-25 15:13:32 on dist-test-slave-p468"
server_key: "6a585b49af4fefef266a92b410e4b3c7"
server_key_iv: "00fa0f60d15900959405f90eacff9b10"
server_key_version: "encryptionkey@0"
I20260925 15:13:32.454164 18522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/ts-0/wal/instance:
uuid: "372093292e564acaae5d8d6370b2a26c"
format_stamp: "Formatted at 2026-09-25 15:13:32 on dist-test-slave-p468"
server_key: "6a585b49af4fefef266a92b410e4b3c7"
server_key_iv: "00fa0f60d15900959405f90eacff9b10"
server_key_version: "encryptionkey@0"
I20260925 15:13:32.459131 18522 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260925 15:13:32.462482 18537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:32.463802 18522 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 15:13:32.463969 18522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/ts-0/wal
uuid: "372093292e564acaae5d8d6370b2a26c"
format_stamp: "Formatted at 2026-09-25 15:13:32 on dist-test-slave-p468"
server_key: "6a585b49af4fefef266a92b410e4b3c7"
server_key_iv: "00fa0f60d15900959405f90eacff9b10"
server_key_version: "encryptionkey@0"
I20260925 15:13:32.464164 18522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:13:32.475606 18522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:13:32.479647 18522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:13:32.479923 18522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:13:32.480918 18522 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:13:32.482178 18522 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:13:32.482366 18522 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:32.482542 18522 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:13:32.482652 18522 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:32.515358 18522 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.1:46055
I20260925 15:13:32.515398 18649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.1:46055 every 8 connection(s)
I20260925 15:13:32.516569 18522 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/ts-0/data/info.pb
I20260925 15:13:32.520251 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 18522
I20260925 15:13:32.520452 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/ts-0/wal/instance
I20260925 15:13:32.521224 12320 external_mini_cluster.cc:1428] Setting key 407271638565c5c50c40b89e3ace99ed
Sep 25 15:13:32 dist-test-slave-p468 krb5kdc[18415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349212, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.8.1@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM
I20260925 15:13:32.537714 18652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:32.519199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.1:40293 (local address 127.12.8.62:44383)
0925 15:13:32.519648 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:32.519654 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:32.520755 (+ 1101us) server_negotiation.cc:407] Connection header received
0925 15:13:32.522949 (+ 2194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:32.522954 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:32.523039 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:32.523127 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:32.524585 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:32.525753 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:32.527344 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:32.528060 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:32.528200 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:32.531998 (+ 3798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:32.532003 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:32.532006 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:32.532036 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:32.534719 (+ 2683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:32.535415 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:32.535423 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:32.535428 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:32.535587 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:32.536040 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:32.536045 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:32.536049 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:32.536307 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:32.536458 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:32.537089 (+ 631us) server_negotiation.cc:299] Negotiation successful
0925 15:13:32.537307 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":216,"threads_started":1}
I20260925 15:13:32.538643 18653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:32.519750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:44383 (local address 127.12.8.1:40293)
0925 15:13:32.520612 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:32.520649 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:32.522663 (+ 2014us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:32.523375 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:32.523382 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:32.523869 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:13:32.524417 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:32.524433 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:32.525884 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:32.525888 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:32.527204 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:32.527219 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:32.528260 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:32.528264 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:32.528449 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:32.529174 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:13:32.529210 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:13:32.531789 (+ 2579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:13:32.534851 (+ 3062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:13:32.534859 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:13:32.534875 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:13:32.535278 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:13:32.535696 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:13:32.535702 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:13:32.535706 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:13:32.535930 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:13:32.536430 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:13:32.536437 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:13:32.536752 (+ 315us) client_negotiation.cc:769] Sending connection context
0925 15:13:32.537004 (+ 252us) client_negotiation.cc:240] Negotiation successful
0925 15:13:32.537507 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":337,"threads_started":1}
I20260925 15:13:32.540764 18650 heartbeater.cc:347] Connected to a master server at 127.12.8.62:44383
I20260925 15:13:32.541133 18650 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for oryx/127.12.8.2@KRBTEST.COM; defaulting to no policy
I20260925 15:13:32.541985 18650 heartbeater.cc:511] Master 127.12.8.62:44383 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.8.2@KRBTEST.COM" created.
I20260925 15:13:32.544164 18460 ts_manager.cc:194] Registered new tserver with Master: 372093292e564acaae5d8d6370b2a26c (127.12.8.1:46055)
I20260925 15:13:32.546295 18460 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.8.1@KRBTEST.COM'} at 127.12.8.1:40293
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.8.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/krb5kdc/oryx_127.12.8.2.keytab.
Entry for principal oryx/127.12.8.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/krb5kdc/oryx_127.12.8.2.keytab.
I20260925 15:13:32.567770 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.2:0
--local_ip_for_outbound_sockets=127.12.8.2
--webserver_interface=127.12.8.2
--webserver_port=0
--tserver_master_addrs=127.12.8.62:44383
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/krb5kdc/oryx_127.12.8.2.keytab
--principal=oryx/127.12.8.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:36421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:13:32.890115 18656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:13:32.890502 18656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:13:32.890681 18656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:13:32.897994 18656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:13:32.898137 18656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:13:32.898326 18656 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.2
I20260925 15:13:32.908025 18656 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:36421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/krb5kdc/oryx_127.12.8.2.keytab
--principal=oryx/127.12.8.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.8.2
--webserver_port=0
--tserver_master_addrs=127.12.8.62:44383
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.2
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:13:32.909302 18656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:13:32.910619 18656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:13:32.923287 18661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:32.923339 18662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:32.923591 18664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:13:32.924373 18656 server_base.cc:1034] running on GCE node
I20260925 15:13:32.925027 18656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:13:32.925943 18656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:13:32.927206 18656 hybrid_clock.cc:648] HybridClock initialized: now 1790349212927098 us; error 76 us; skew 500 ppm
I20260925 15:13:32.927603 18656 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:13:32 dist-test-slave-p468 krb5kdc[18415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349212, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.8.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:13:32.930768 18656 init.cc:377] Logged in from keytab as oryx/127.12.8.2@KRBTEST.COM (short username oryx)
I20260925 15:13:32.932194 18656 webserver.cc:460] Webserver started at http://127.12.8.2:41793/ using document root <none> and password file <none>
I20260925 15:13:32.932809 18656 fs_manager.cc:359] Metadata directory not provided
I20260925 15:13:32.932906 18656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:13:32.933128 18656 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:13:32.935935 18656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/ts-1/data/instance:
uuid: "ce72ef070a95404f8dbb29d80d7cff69"
format_stamp: "Formatted at 2026-09-25 15:13:32 on dist-test-slave-p468"
server_key: "5266384aaf8008ac737a5dfe4010ef1f"
server_key_iv: "96c15a2a7e04a251d48a29691874d141"
server_key_version: "encryptionkey@0"
I20260925 15:13:32.936798 18656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/ts-1/wal/instance:
uuid: "ce72ef070a95404f8dbb29d80d7cff69"
format_stamp: "Formatted at 2026-09-25 15:13:32 on dist-test-slave-p468"
server_key: "5266384aaf8008ac737a5dfe4010ef1f"
server_key_iv: "96c15a2a7e04a251d48a29691874d141"
server_key_version: "encryptionkey@0"
I20260925 15:13:32.942976 18656 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260925 15:13:32.947074 18671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:32.948601 18656 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.003s
I20260925 15:13:32.948751 18656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/ts-1/wal
uuid: "ce72ef070a95404f8dbb29d80d7cff69"
format_stamp: "Formatted at 2026-09-25 15:13:32 on dist-test-slave-p468"
server_key: "5266384aaf8008ac737a5dfe4010ef1f"
server_key_iv: "96c15a2a7e04a251d48a29691874d141"
server_key_version: "encryptionkey@0"
I20260925 15:13:32.948943 18656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:13:32.966775 18656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:13:32.971099 18656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:13:32.971343 18656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:13:32.972272 18656 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:13:32.973538 18656 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:13:32.973615 18656 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:32.973711 18656 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:13:32.973762 18656 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:33.007849 18656 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.2:35849
I20260925 15:13:33.007903 18783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.2:35849 every 8 connection(s)
I20260925 15:13:33.009204 18656 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/ts-1/data/info.pb
I20260925 15:13:33.016801 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 18656
I20260925 15:13:33.017020 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/ts-1/wal/instance
I20260925 15:13:33.017576 12320 external_mini_cluster.cc:1428] Setting key 784c126085aa2286595077d46a3ac535
Sep 25 15:13:33 dist-test-slave-p468 krb5kdc[18415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349212, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.8.2@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM
I20260925 15:13:33.030992 18652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:33.011949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.2:43583 (local address 127.12.8.62:44383)
0925 15:13:33.012140 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:33.012145 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:33.013328 (+ 1183us) server_negotiation.cc:407] Connection header received
0925 15:13:33.015206 (+ 1878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:33.015211 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:33.015285 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:33.015384 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:33.016856 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:33.017956 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:33.019849 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:33.020519 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:33.020674 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:33.024598 (+ 3924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:33.024607 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:33.024615 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:33.024661 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:33.027548 (+ 2887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:33.028599 (+ 1051us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:33.028602 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:33.028605 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:33.028717 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:33.029233 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:33.029236 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:33.029237 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:33.029478 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:33.029598 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:33.030579 (+ 981us) server_negotiation.cc:299] Negotiation successful
0925 15:13:33.030749 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 15:13:33.032670 18786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:33.012452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:44383 (local address 127.12.8.2:43583)
0925 15:13:33.013159 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:33.013195 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:33.014940 (+ 1745us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:33.015609 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:33.015617 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:33.016106 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:13:33.016662 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:33.016679 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:33.018161 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:33.018169 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:33.019713 (+ 1544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:33.019725 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:33.020731 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:33.020736 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:33.020923 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:33.021669 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:13:33.021693 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:13:33.024354 (+ 2661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:13:33.027695 (+ 3341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:13:33.027708 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:13:33.027729 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:13:33.028468 (+ 739us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:13:33.028824 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:13:33.028833 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:13:33.028840 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:13:33.029120 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:13:33.029611 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:13:33.029623 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:13:33.030117 (+ 494us) client_negotiation.cc:769] Sending connection context
0925 15:13:33.030469 (+ 352us) client_negotiation.cc:240] Negotiation successful
0925 15:13:33.031107 (+ 638us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":294,"threads_started":1}
I20260925 15:13:33.034288 18784 heartbeater.cc:347] Connected to a master server at 127.12.8.62:44383
I20260925 15:13:33.034603 18784 heartbeater.cc:464] Registering TS with master...
I20260925 15:13:33.035341 18784 heartbeater.cc:511] Master 127.12.8.62:44383 requested a full tablet report, sending...
I20260925 15:13:33.037271 18460 ts_manager.cc:194] Registered new tserver with Master: ce72ef070a95404f8dbb29d80d7cff69 (127.12.8.2:35849)
I20260925 15:13:33.038704 18460 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.8.2@KRBTEST.COM'} at 127.12.8.2:43583
WARNING: no policy specified for oryx/127.12.8.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.8.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.8.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/krb5kdc/oryx_127.12.8.3.keytab.
Entry for principal oryx/127.12.8.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/krb5kdc/oryx_127.12.8.3.keytab.
I20260925 15:13:33.065447 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.3:0
--local_ip_for_outbound_sockets=127.12.8.3
--webserver_interface=127.12.8.3
--webserver_port=0
--tserver_master_addrs=127.12.8.62:44383
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/krb5kdc/oryx_127.12.8.3.keytab
--principal=oryx/127.12.8.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:36421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:13:33.379098 18789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:13:33.379423 18789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:13:33.379585 18789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:13:33.386873 18789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:13:33.386971 18789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:13:33.387149 18789 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.3
I20260925 15:13:33.396669 18789 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:36421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/krb5kdc/oryx_127.12.8.3.keytab
--principal=oryx/127.12.8.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.8.3
--webserver_port=0
--tserver_master_addrs=127.12.8.62:44383
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.3
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:13:33.397928 18789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:13:33.399380 18789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:13:33.411974 18794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:33.412011 18795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:33.412133 18797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:13:33.413240 18789 server_base.cc:1034] running on GCE node
I20260925 15:13:33.413892 18789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:13:33.415530 18789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:13:33.416733 18789 hybrid_clock.cc:648] HybridClock initialized: now 1790349213416662 us; error 68 us; skew 500 ppm
I20260925 15:13:33.417063 18789 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:13:33 dist-test-slave-p468 krb5kdc[18415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349213, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.8.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:13:33.420490 18789 init.cc:377] Logged in from keytab as oryx/127.12.8.3@KRBTEST.COM (short username oryx)
I20260925 15:13:33.421962 18789 webserver.cc:460] Webserver started at http://127.12.8.3:44911/ using document root <none> and password file <none>
I20260925 15:13:33.422727 18789 fs_manager.cc:359] Metadata directory not provided
I20260925 15:13:33.422858 18789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:13:33.423166 18789 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:13:33.426110 18789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/ts-2/data/instance:
uuid: "998c5926ac2741278b4a9c80cd3f8df5"
format_stamp: "Formatted at 2026-09-25 15:13:33 on dist-test-slave-p468"
server_key: "b0c0c690b2a3a4dc61f5e9277b13b03c"
server_key_iv: "eda382e2585032d5f393a671dabac984"
server_key_version: "encryptionkey@0"
I20260925 15:13:33.426981 18789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/ts-2/wal/instance:
uuid: "998c5926ac2741278b4a9c80cd3f8df5"
format_stamp: "Formatted at 2026-09-25 15:13:33 on dist-test-slave-p468"
server_key: "b0c0c690b2a3a4dc61f5e9277b13b03c"
server_key_iv: "eda382e2585032d5f393a671dabac984"
server_key_version: "encryptionkey@0"
I20260925 15:13:33.432061 18789 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 15:13:33.435599 18804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:33.436889 18789 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 15:13:33.437062 18789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/ts-2/wal
uuid: "998c5926ac2741278b4a9c80cd3f8df5"
format_stamp: "Formatted at 2026-09-25 15:13:33 on dist-test-slave-p468"
server_key: "b0c0c690b2a3a4dc61f5e9277b13b03c"
server_key_iv: "eda382e2585032d5f393a671dabac984"
server_key_version: "encryptionkey@0"
I20260925 15:13:33.437289 18789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:13:33.448038 18789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:13:33.451774 18789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:13:33.452050 18789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:13:33.452948 18789 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:13:33.454322 18789 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:13:33.454425 18789 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:33.454578 18789 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:13:33.454644 18789 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:33.487593 18789 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.3:40103
I20260925 15:13:33.487639 18916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.3:40103 every 8 connection(s)
I20260925 15:13:33.488893 18789 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/ts-2/data/info.pb
I20260925 15:13:33.493130 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 18789
I20260925 15:13:33.493310 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/ts-2/wal/instance
I20260925 15:13:33.493741 12320 external_mini_cluster.cc:1428] Setting key 9aeaecba98898ef64bdfc30d51399a16
Sep 25 15:13:33 dist-test-slave-p468 krb5kdc[18415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349213, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.8.3@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM
I20260925 15:13:33.507802 18652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:33.491347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.3:60357 (local address 127.12.8.62:44383)
0925 15:13:33.491510 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:33.491514 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:33.492622 (+ 1108us) server_negotiation.cc:407] Connection header received
0925 15:13:33.494627 (+ 2005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:33.494632 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:33.494716 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:33.494838 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:33.496271 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:33.497104 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:33.498680 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:33.499122 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:33.499255 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:33.502548 (+ 3293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:33.502553 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:33.502556 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:33.502585 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:33.504781 (+ 2196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:33.505448 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:33.505454 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:33.505457 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:33.505578 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:33.505982 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:33.505987 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:33.505990 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:33.506326 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:33.506473 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:33.507205 (+ 732us) server_negotiation.cc:299] Negotiation successful
0925 15:13:33.507410 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 15:13:33.508811 18919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:33.491775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:44383 (local address 127.12.8.3:60357)
0925 15:13:33.492449 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:33.492492 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:33.494331 (+ 1839us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:33.495085 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:33.495093 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:33.495570 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:13:33.496109 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:33.496126 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:33.497251 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:33.497257 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:33.498523 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:33.498554 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:33.499334 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:33.499343 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:33.499633 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:33.500235 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:13:33.500255 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:13:33.502338 (+ 2083us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:13:33.504916 (+ 2578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:13:33.504924 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:13:33.504938 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:13:33.505328 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:13:33.505684 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:13:33.505690 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:13:33.505693 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:13:33.505866 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:13:33.506444 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:13:33.506451 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:13:33.506786 (+ 335us) client_negotiation.cc:769] Sending connection context
0925 15:13:33.507043 (+ 257us) client_negotiation.cc:240] Negotiation successful
0925 15:13:33.507507 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":267,"threads_started":1}
I20260925 15:13:33.510020 18917 heartbeater.cc:347] Connected to a master server at 127.12.8.62:44383
I20260925 15:13:33.510382 18917 heartbeater.cc:464] Registering TS with master...
I20260925 15:13:33.511068 18917 heartbeater.cc:511] Master 127.12.8.62:44383 requested a full tablet report, sending...
I20260925 15:13:33.512411 18460 ts_manager.cc:194] Registered new tserver with Master: 998c5926ac2741278b4a9c80cd3f8df5 (127.12.8.3:40103)
I20260925 15:13:33.513209 18460 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.8.3@KRBTEST.COM'} at 127.12.8.3:60357
I20260925 15:13:33.520715 12320 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.12.8.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.8.61@KRBTEST.COM" created.
I20260925 15:13:33.549147 18650 heartbeater.cc:503] Master 127.12.8.62:44383 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.8.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/krb5kdc/oryx_127.12.8.61.keytab.
Entry for principal oryx/127.12.8.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/krb5kdc/oryx_127.12.8.61.keytab.
I20260925 15:13:33.565616 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.61:35001
--webserver_interface=127.12.8.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/krb5kdc/oryx_127.12.8.61.keytab
--principal=oryx/127.12.8.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:36421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.62:44383,127.12.8.61:35001
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:13:33.884169 18922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:13:33.884442 18922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:13:33.884528 18922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:13:33.891630 18922 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:13:33.891737 18922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:13:33.891793 18922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:13:33.891847 18922 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:13:33.891891 18922 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:13:33.901548 18922 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:36421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.12.8.62:44383,127.12.8.61:35001
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/krb5kdc/oryx_127.12.8.61.keytab
--principal=oryx/127.12.8.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.61:35001
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.12.8.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
W20260925 15:13:33.901840 18922 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.12.8.62:44383,127.12.8.61:35001'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260925 15:13:33.902829 18922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:13:33.904053 18922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:13:33.915098 18927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:33.915217 18930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:33.915148 18928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:13:33.915867 18922 server_base.cc:1034] running on GCE node
I20260925 15:13:33.916594 18922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:13:33.917884 18922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:13:33.919135 18922 hybrid_clock.cc:648] HybridClock initialized: now 1790349213919068 us; error 57 us; skew 500 ppm
I20260925 15:13:33.919452 18922 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:13:33 dist-test-slave-p468 krb5kdc[18415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349213, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.8.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:13:33.922442 18922 init.cc:377] Logged in from keytab as oryx/127.12.8.61@KRBTEST.COM (short username oryx)
I20260925 15:13:33.923821 18922 webserver.cc:460] Webserver started at http://127.12.8.61:44505/ using document root <none> and password file <none>
I20260925 15:13:33.924491 18922 fs_manager.cc:359] Metadata directory not provided
I20260925 15:13:33.924614 18922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:13:33.924922 18922 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:13:33.927893 18922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/master-1/data/instance:
uuid: "ea8b382304f143f4b4a3dca87c09b7e4"
format_stamp: "Formatted at 2026-09-25 15:13:33 on dist-test-slave-p468"
server_key: "89b50309b9c08556a9d17055e1a6fd8d"
server_key_iv: "6fc3f259d27687abcbc004ba2da353be"
server_key_version: "encryptionkey@0"
I20260925 15:13:33.928653 18922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/master-1/wal/instance:
uuid: "ea8b382304f143f4b4a3dca87c09b7e4"
format_stamp: "Formatted at 2026-09-25 15:13:33 on dist-test-slave-p468"
server_key: "89b50309b9c08556a9d17055e1a6fd8d"
server_key_iv: "6fc3f259d27687abcbc004ba2da353be"
server_key_version: "encryptionkey@0"
I20260925 15:13:33.933472 18922 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 15:13:33.936872 18937 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:33.938517 18922 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:13:33.938688 18922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/master-1/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/master-1/wal
uuid: "ea8b382304f143f4b4a3dca87c09b7e4"
format_stamp: "Formatted at 2026-09-25 15:13:33 on dist-test-slave-p468"
server_key: "89b50309b9c08556a9d17055e1a6fd8d"
server_key_iv: "6fc3f259d27687abcbc004ba2da353be"
server_key_version: "encryptionkey@0"
I20260925 15:13:33.938894 18922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:13:33.954809 18922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:13:33.958599 18922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:13:33.958807 18922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:13:33.977888 18922 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.61:35001
I20260925 15:13:33.977907 18988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.61:35001 every 8 connection(s)
I20260925 15:13:33.979331 18922 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/master-1/data/info.pb
I20260925 15:13:33.983162 18989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:13:33.983304 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 18922
I20260925 15:13:33.992396 18989 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.8.62" port: 44383 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:13:33 dist-test-slave-p468 krb5kdc[18415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349211, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.8.61@KRBTEST.COM
Sep 25 15:13:33 dist-test-slave-p468 krb5kdc[18415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349213, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.8.61@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM
I20260925 15:13:34.009807 18992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:33.985795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49438 (local address 127.12.8.61:35001)
0925 15:13:33.986740 (+ 945us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:33.986755 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:33.987216 (+ 461us) server_negotiation.cc:407] Connection header received
0925 15:13:33.989429 (+ 2213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:33.989457 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:33.989965 (+ 508us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:33.990412 (+ 447us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:33.991851 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:33.993867 (+ 2016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:33.995259 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:33.996339 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:33.996503 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:34.001455 (+ 4952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:34.001462 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:34.001473 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:34.001524 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:34.004814 (+ 3290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:34.005610 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:34.005619 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:34.005629 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:34.005803 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:34.006241 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:34.006246 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:34.006250 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:34.006973 (+ 723us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:34.007282 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:34.008010 (+ 728us) server_negotiation.cc:299] Negotiation successful
0925 15:13:34.008513 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":692,"thread_start_us":421,"threads_started":1}
I20260925 15:13:34.011076 18652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:33.995367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60638 (local address 127.12.8.62:44383)
0925 15:13:33.995528 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:33.995532 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:33.996250 (+ 718us) server_negotiation.cc:407] Connection header received
0925 15:13:33.996467 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:33.996473 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:33.996553 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:33.996669 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:33.997667 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:33.998873 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:34.000501 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:34.000918 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:34.001015 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:34.004405 (+ 3390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:34.004411 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:34.004416 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:34.004457 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:34.007865 (+ 3408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:34.008827 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:34.008834 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:34.008837 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:34.008975 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:34.009422 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:34.009430 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:34.009435 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:34.009746 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:34.009835 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:34.010527 (+ 692us) server_negotiation.cc:299] Negotiation successful
0925 15:13:34.010739 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 15:13:34.011116 18993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:33.995412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:44383 (local address 127.0.0.1:60638)
0925 15:13:33.996101 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:33.996156 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:33.996328 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:33.996735 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:33.996739 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:33.997281 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:13:33.997503 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:33.997510 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:33.999009 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:33.999013 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:34.000337 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:34.000355 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:34.001144 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:34.001151 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:34.001344 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:34.001987 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:13:34.002012 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:13:34.004213 (+ 2201us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:13:34.008025 (+ 3812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:13:34.008036 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:13:34.008043 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:13:34.008703 (+ 660us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:13:34.009095 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:13:34.009100 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:13:34.009104 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:13:34.009284 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:13:34.009973 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:13:34.009983 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:13:34.010168 (+ 185us) client_negotiation.cc:769] Sending connection context
0925 15:13:34.010409 (+ 241us) client_negotiation.cc:240] Negotiation successful
0925 15:13:34.010719 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":351,"threads_started":1}
I20260925 15:13:34.012992 18989 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.8.61" port: 35001 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:13:34 dist-test-slave-p468 krb5kdc[18415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349213, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.8.61@KRBTEST.COM for oryx/127.12.8.61@KRBTEST.COM
I20260925 15:13:34.025701 18993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:34.013788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.61:35001 (local address 127.0.0.1:49450)
0925 15:13:34.014092 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:34.014106 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:34.014239 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:34.014506 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:34.014509 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:34.014806 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:13:34.014940 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:34.014947 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:34.015813 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:34.015816 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:34.016655 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:34.016665 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:34.017636 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:34.017643 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:34.017869 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:34.018457 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:13:34.018473 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:13:34.020449 (+ 1976us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:13:34.023390 (+ 2941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:13:34.023393 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:13:34.023396 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:13:34.023763 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:13:34.024130 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:13:34.024134 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:13:34.024136 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:13:34.024251 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:13:34.024678 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:13:34.024681 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:13:34.024798 (+ 117us) client_negotiation.cc:769] Sending connection context
0925 15:13:34.024903 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 15:13:34.025208 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199}
I20260925 15:13:34.025852 18992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:34.013788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49450 (local address 127.12.8.61:35001)
0925 15:13:34.013962 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:34.013970 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:34.014175 (+ 205us) server_negotiation.cc:407] Connection header received
0925 15:13:34.014342 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:34.014346 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:34.014397 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:34.014472 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:34.015051 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:34.015712 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:34.016792 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:34.017461 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:34.017622 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:34.020620 (+ 2998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:34.020624 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:34.020627 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:34.020654 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:34.023266 (+ 2612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:34.023881 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:34.023885 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:34.023887 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:34.024015 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:34.024342 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:34.024345 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:34.024347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:34.024577 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:34.024694 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:34.025057 (+ 363us) server_negotiation.cc:299] Negotiation successful
0925 15:13:34.025407 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":71,"server-negotiator.queue_time_us":57}
I20260925 15:13:34.036710 18989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea8b382304f143f4b4a3dca87c09b7e4: Bootstrap starting.
I20260925 15:13:34.041452 18784 heartbeater.cc:503] Master 127.12.8.62:44383 was elected leader, sending a full tablet report...
I20260925 15:13:34.042055 18989 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea8b382304f143f4b4a3dca87c09b7e4: Neither blocks nor log segments found. Creating new log.
I20260925 15:13:34.043418 18989 log.cc:824] T 00000000000000000000000000000000 P ea8b382304f143f4b4a3dca87c09b7e4: Log is configured to *not* fsync() on all Append() calls
I20260925 15:13:34.046859 18989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea8b382304f143f4b4a3dca87c09b7e4: No bootstrap required, opened a new log
I20260925 15:13:34.058082 18989 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea8b382304f143f4b4a3dca87c09b7e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1233ca1ad3745eca6025d08fc59d7f3" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 44383 } } peers { permanent_uuid: "ea8b382304f143f4b4a3dca87c09b7e4" member_type: VOTER last_known_addr { host: "127.12.8.61" port: 35001 } }
I20260925 15:13:34.058509 18989 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea8b382304f143f4b4a3dca87c09b7e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:13:34.058624 18989 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea8b382304f143f4b4a3dca87c09b7e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea8b382304f143f4b4a3dca87c09b7e4, State: Initialized, Role: FOLLOWER
I20260925 15:13:34.059311 18989 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea8b382304f143f4b4a3dca87c09b7e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1233ca1ad3745eca6025d08fc59d7f3" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 44383 } } peers { permanent_uuid: "ea8b382304f143f4b4a3dca87c09b7e4" member_type: VOTER last_known_addr { host: "127.12.8.61" port: 35001 } }
I20260925 15:13:34.060307 18995 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea8b382304f143f4b4a3dca87c09b7e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1233ca1ad3745eca6025d08fc59d7f3" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 44383 } } peers { permanent_uuid: "ea8b382304f143f4b4a3dca87c09b7e4" member_type: VOTER last_known_addr { host: "127.12.8.61" port: 35001 } } }
I20260925 15:13:34.060681 18995 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea8b382304f143f4b4a3dca87c09b7e4 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:13:34.061725 18989 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea8b382304f143f4b4a3dca87c09b7e4 [sys.catalog]: configured and running, proceeding with master startup.
W20260925 15:13:34.072609 19006 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ea8b382304f143f4b4a3dca87c09b7e4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:13:34.072786 19006 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:13:34.074784 18922 master_runner.cc:428] Detected that this master ea8b382304f143f4b4a3dca87c09b7e4 is joining an existing cluster
I20260925 15:13:34.074890 18922 master_runner.cc:432] Initiating AddMaster RPC to add 127.12.8.61:35001
I20260925 15:13:34.095305 19012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:34.079165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.61:35001 (local address 127.0.0.1:49456)
0925 15:13:34.079838 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:34.079859 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:34.080003 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:34.080406 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:34.080414 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:34.081193 (+ 779us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:13:34.081649 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:34.081666 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:34.083321 (+ 1655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:34.083328 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:34.084613 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:34.084625 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:34.085848 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:34.085853 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:34.086048 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:34.086868 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:13:34.086892 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:13:34.087976 (+ 1084us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:13:34.092142 (+ 4166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:13:34.092149 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:13:34.092156 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:13:34.092671 (+ 515us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:13:34.093128 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:13:34.093135 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:13:34.093139 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:13:34.093301 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:13:34.093974 (+ 673us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:13:34.093980 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:13:34.094151 (+ 171us) client_negotiation.cc:769] Sending connection context
0925 15:13:34.094411 (+ 260us) client_negotiation.cc:240] Negotiation successful
0925 15:13:34.094715 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"mutex_wait_us":22,"spinlock_wait_cycles":21888,"thread_start_us":497,"threads_started":1}
I20260925 15:13:34.095590 18992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:34.079663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49456 (local address 127.12.8.61:35001)
0925 15:13:34.079861 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:34.079868 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:34.079926 (+ 58us) server_negotiation.cc:407] Connection header received
0925 15:13:34.080159 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:34.080165 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:34.080262 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:34.080394 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:34.081796 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:34.083153 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:34.084758 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:34.085678 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:34.085834 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:34.088192 (+ 2358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:34.088198 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:34.088203 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:34.088236 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:34.091975 (+ 3739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:34.092806 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:34.092811 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:34.092814 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:34.092972 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:34.093409 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:34.093414 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:34.093417 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:34.093748 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:34.093885 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:34.094533 (+ 648us) server_negotiation.cc:299] Negotiation successful
0925 15:13:34.094901 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":99,"server-negotiator.queue_time_us":70}
I20260925 15:13:34.099891 18652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:34.079065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60654 (local address 127.12.8.62:44383)
0925 15:13:34.079636 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:34.079645 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:34.086842 (+ 7197us) server_negotiation.cc:407] Connection header received
0925 15:13:34.087097 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:34.087104 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:34.087192 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:34.087270 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:34.088711 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:34.089608 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:34.091233 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:34.091657 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:34.091783 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:34.093503 (+ 1720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:34.093509 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:34.093514 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:34.093554 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:34.097146 (+ 3592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:34.097753 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:34.097759 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:34.097762 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:34.097891 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:34.098360 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:34.098368 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:34.098371 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:34.098669 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:34.098822 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:34.099285 (+ 463us) server_negotiation.cc:299] Negotiation successful
0925 15:13:34.099527 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437}
I20260925 15:13:34.099949 19013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:34.080243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:44383 (local address 127.0.0.1:60654)
0925 15:13:34.086729 (+ 6486us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:34.086758 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:34.086949 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:34.087538 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:34.087549 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:34.088288 (+ 739us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:13:34.088546 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:34.088553 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:34.089762 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:34.089768 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:34.091074 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:34.091086 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:34.091857 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:34.091863 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:34.092015 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:34.092579 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:13:34.092602 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:13:34.093329 (+ 727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:13:34.097275 (+ 3946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:13:34.097281 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:13:34.097284 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:13:34.097620 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:13:34.098024 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:13:34.098028 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:13:34.098032 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:13:34.098165 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:13:34.098788 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:13:34.098793 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:13:34.098983 (+ 190us) client_negotiation.cc:769] Sending connection context
0925 15:13:34.099134 (+ 151us) client_negotiation.cc:240] Negotiation successful
0925 15:13:34.099512 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6329,"thread_start_us":6129,"threads_started":1}
Sep 25 15:13:34 dist-test-slave-p468 krb5kdc[18415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349211, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.8.62@KRBTEST.COM for oryx/127.12.8.61@KRBTEST.COM
I20260925 15:13:34.120460 18992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:34.105683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49472 (local address 127.12.8.61:35001)
0925 15:13:34.105846 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:34.105850 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:34.106477 (+ 627us) server_negotiation.cc:407] Connection header received
0925 15:13:34.106768 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:34.106772 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:34.106856 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:34.106958 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:34.107959 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:34.108710 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:34.110488 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:34.110939 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:34.111087 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:34.114473 (+ 3386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:34.114478 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:34.114481 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:34.114538 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:34.117671 (+ 3133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:34.118452 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:34.118459 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:34.118468 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:34.118599 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:34.118984 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:34.118993 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:34.118995 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:34.119262 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:34.119415 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:34.119875 (+ 460us) server_negotiation.cc:299] Negotiation successful
0925 15:13:34.120118 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 15:13:34.120662 19014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:34.105714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.61:35001 (local address 127.0.0.1:49472)
0925 15:13:34.106354 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:34.106392 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:34.106574 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:34.107002 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:34.107010 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:34.107633 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:13:34.107818 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:34.107827 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:34.108838 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:34.108846 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:34.110343 (+ 1497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:34.110361 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:34.111110 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:34.111116 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:34.111308 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:34.111923 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:13:34.111941 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:13:34.114248 (+ 2307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:13:34.117815 (+ 3567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:13:34.117828 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:13:34.117837 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:13:34.118327 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:13:34.118714 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:13:34.118719 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:13:34.118722 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:13:34.118869 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:13:34.119395 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:13:34.119404 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:13:34.119592 (+ 188us) client_negotiation.cc:769] Sending connection context
0925 15:13:34.119778 (+ 186us) client_negotiation.cc:240] Negotiation successful
0925 15:13:34.120107 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":264,"threads_started":1}
I20260925 15:13:34.122552 18460 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.12.8.61:35001: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "ea8b382304f143f4b4a3dca87c09b7e4"
member_type: NON_VOTER
last_known_addr {
host: "127.12.8.61"
port: 35001
}
attrs {
promote: true
}
}
dest_uuid: "c1233ca1ad3745eca6025d08fc59d7f3"
cas_config_opid_index: -1
I20260925 15:13:34.123502 18460 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1233ca1ad3745eca6025d08fc59d7f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c1233ca1ad3745eca6025d08fc59d7f3" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 44383 } } peers { permanent_uuid: "ea8b382304f143f4b4a3dca87c09b7e4" member_type: NON_VOTER last_known_addr { host: "127.12.8.61" port: 35001 } attrs { promote: true } }
I20260925 15:13:34.126739 19017 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c1233ca1ad3745eca6025d08fc59d7f3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER ea8b382304f143f4b4a3dca87c09b7e4 (127.12.8.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c1233ca1ad3745eca6025d08fc59d7f3" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 44383 } } peers { permanent_uuid: "ea8b382304f143f4b4a3dca87c09b7e4" member_type: NON_VOTER last_known_addr { host: "127.12.8.61" port: 35001 } attrs { promote: true } } }
I20260925 15:13:34.127576 18964 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea8b382304f143f4b4a3dca87c09b7e4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:13:34.130990 19017 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.12.8.61:35001
I20260925 15:13:34.131156 18964 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ea8b382304f143f4b4a3dca87c09b7e4 [term 1 FOLLOWER]: Refusing update from remote peer c1233ca1ad3745eca6025d08fc59d7f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260925 15:13:34.131548 19017 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1233ca1ad3745eca6025d08fc59d7f3 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c1233ca1ad3745eca6025d08fc59d7f3" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c1233ca1ad3745eca6025d08fc59d7f3" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 44383 } } peers { permanent_uuid: "ea8b382304f143f4b4a3dca87c09b7e4" member_type: NON_VOTER last_known_addr { host: "127.12.8.61" port: 35001 } attrs { promote: true } } }
I20260925 15:13:34.131807 18922 master.cc:537] Master@127.12.8.61:35001 shutting down...
I20260925 15:13:34.131912 19017 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1233ca1ad3745eca6025d08fc59d7f3 [sys.catalog]: This master's current role is: LEADER
I20260925 15:13:34.133765 19018 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c1233ca1ad3745eca6025d08fc59d7f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea8b382304f143f4b4a3dca87c09b7e4" member_type: NON_VOTER last_known_addr { host: "127.12.8.61" port: 35001 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 15:13:34.137068 18444 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260925 15:13:34.147094 18444 consensus_peers.cc:487] T 00000000000000000000000000000000 P c1233ca1ad3745eca6025d08fc59d7f3 -> Peer ea8b382304f143f4b4a3dca87c09b7e4 (127.12.8.61:35001): Couldn't send request to peer ea8b382304f143f4b4a3dca87c09b7e4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
WARNING: no policy specified for oryx/127.12.8.60@KRBTEST.COM; defaulting to no policy
I20260925 15:13:34.149257 18922 raft_consensus.cc:2229] T 00000000000000000000000000000000 P ea8b382304f143f4b4a3dca87c09b7e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:13:34.149667 18922 raft_consensus.cc:2258] T 00000000000000000000000000000000 P ea8b382304f143f4b4a3dca87c09b7e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:13:34.149865 18922 tablet_replica.cc:330] T 00000000000000000000000000000000 P ea8b382304f143f4b4a3dca87c09b7e4: stopping tablet replica
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.8.60@KRBTEST.COM" created.
I20260925 15:13:34.158771 18922 master.cc:552] Master@127.12.8.61:35001 shutdown complete.
I20260925 15:13:34.158869 18922 master_runner.cc:305] Clearing existing system tablet
I20260925 15:13:34.159216 18922 fs_manager.cc:359] Metadata directory not provided
I20260925 15:13:34.159322 18922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:13:34.161521 18922 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:13:34.163178 19023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:34.163470 18922 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:13:34.163743 18922 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:13:34.163833 18922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/master-1/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/master-1/wal
uuid: "ea8b382304f143f4b4a3dca87c09b7e4"
format_stamp: "Formatted at 2026-09-25 15:13:33 on dist-test-slave-p468"
server_key: "89b50309b9c08556a9d17055e1a6fd8d"
server_key_iv: "6fc3f259d27687abcbc004ba2da353be"
server_key_version: "encryptionkey@0"
I20260925 15:13:34.169631 18922 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P ea8b382304f143f4b4a3dca87c09b7e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:13:34.174540 18922 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P ea8b382304f143f4b4a3dca87c09b7e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 15:13:34.174741 18922 log.cc:1196] T 00000000000000000000000000000000 P ea8b382304f143f4b4a3dca87c09b7e4: Deleting WAL directory at /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260925 15:13:34.176012 18922 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P ea8b382304f143f4b4a3dca87c09b7e4: Deleting consensus metadata
I20260925 15:13:34.176294 18922 master_runner.cc:315] Copying system tablet from 127.12.8.62:44383
I20260925 15:13:34.179572 18922 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P ea8b382304f143f4b4a3dca87c09b7e4: tablet copy: Beginning tablet copy session from remote peer at address 127.12.8.62:44383
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.8.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/krb5kdc/oryx_127.12.8.60.keytab.
Entry for principal oryx/127.12.8.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/krb5kdc/oryx_127.12.8.60.keytab.
I20260925 15:13:34.182575 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.60:41857
--webserver_interface=127.12.8.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/krb5kdc/oryx_127.12.8.60.keytab
--principal=oryx/127.12.8.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:36421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.62:44383,127.12.8.61:35001,127.12.8.60:41857
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:13:34.194329 18652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:34.180451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60660 (local address 127.12.8.62:44383)
0925 15:13:34.180692 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:34.180699 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:34.181041 (+ 342us) server_negotiation.cc:407] Connection header received
0925 15:13:34.181271 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:34.181278 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:34.181374 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:34.181490 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:34.182719 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:34.183846 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:34.185813 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:34.186509 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:34.186642 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:34.189093 (+ 2451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:34.189098 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:34.189101 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:34.189133 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:34.191774 (+ 2641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:34.192406 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:34.192411 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:34.192415 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:34.192543 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:34.192915 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:34.192921 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:34.192924 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:34.193202 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:34.193356 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:34.193725 (+ 369us) server_negotiation.cc:299] Negotiation successful
0925 15:13:34.193933 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 15:13:34.194670 19029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:34.180414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:44383 (local address 127.0.0.1:60660)
0925 15:13:34.180939 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:34.180960 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:34.181110 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:34.181543 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:34.181548 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:34.182101 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:13:34.182539 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:34.182557 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:34.184037 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:34.184046 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:34.185655 (+ 1609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:34.185672 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:34.186677 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:34.186689 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:34.186914 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:34.187665 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:13:34.187694 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:13:34.188875 (+ 1181us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:13:34.191912 (+ 3037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:13:34.191918 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:13:34.191922 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:13:34.192277 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:13:34.192668 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:13:34.192673 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:13:34.192676 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:13:34.192806 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:13:34.193319 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:13:34.193324 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:13:34.193480 (+ 156us) client_negotiation.cc:769] Sending connection context
0925 15:13:34.193629 (+ 149us) client_negotiation.cc:240] Negotiation successful
0925 15:13:34.193938 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":302,"threads_started":1}
I20260925 15:13:34.195837 18480 tablet_copy_service.cc:140] P c1233ca1ad3745eca6025d08fc59d7f3: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer ea8b382304f143f4b4a3dca87c09b7e4 ({username='oryx', principal='oryx/127.12.8.61@KRBTEST.COM'} at 127.0.0.1:60660)
I20260925 15:13:34.196130 18480 tablet_copy_service.cc:161] P c1233ca1ad3745eca6025d08fc59d7f3: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer ea8b382304f143f4b4a3dca87c09b7e4 at {username='oryx', principal='oryx/127.12.8.61@KRBTEST.COM'} at 127.0.0.1:60660: session id = ea8b382304f143f4b4a3dca87c09b7e4-00000000000000000000000000000000
I20260925 15:13:34.199437 18480 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P c1233ca1ad3745eca6025d08fc59d7f3: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 15:13:34.201368 18922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 15:13:34.205847 18922 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P ea8b382304f143f4b4a3dca87c09b7e4: tablet copy: Starting download of 0 data blocks...
I20260925 15:13:34.206131 18922 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P ea8b382304f143f4b4a3dca87c09b7e4: tablet copy: Starting download of 1 WAL segments...
I20260925 15:13:34.208561 18922 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P ea8b382304f143f4b4a3dca87c09b7e4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 15:13:34.211797 18480 tablet_copy_service.cc:342] P c1233ca1ad3745eca6025d08fc59d7f3: Request end of tablet copy session ea8b382304f143f4b4a3dca87c09b7e4-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.12.8.61@KRBTEST.COM'} at 127.0.0.1:60660
I20260925 15:13:34.211983 18480 tablet_copy_service.cc:433] P c1233ca1ad3745eca6025d08fc59d7f3: ending tablet copy session ea8b382304f143f4b4a3dca87c09b7e4-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer ea8b382304f143f4b4a3dca87c09b7e4
I20260925 15:13:34.221511 18922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:13:34.224146 19031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:34.227710 19032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:13:34.232528 18922 server_base.cc:1034] running on GCE node
W20260925 15:13:34.232528 19035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:13:34.233098 18922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:13:34.233695 18922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:13:34.234905 18922 hybrid_clock.cc:648] HybridClock initialized: now 1790349214234846 us; error 38 us; skew 500 ppm
I20260925 15:13:34.235121 18922 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:13:34 dist-test-slave-p468 krb5kdc[18415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349214, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.8.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:13:34.236967 18922 init.cc:377] Logged in from keytab as oryx/127.12.8.61@KRBTEST.COM (short username oryx)
I20260925 15:13:34.239154 18922 webserver.cc:460] Webserver started at http://127.12.8.61:36897/ using document root <none> and password file <none>
I20260925 15:13:34.239573 18922 fs_manager.cc:359] Metadata directory not provided
I20260925 15:13:34.239653 18922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:13:34.242596 18922 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260925 15:13:34.244268 19041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:34.244704 18922 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:13:34.244843 18922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/master-1/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/master-1/wal
uuid: "ea8b382304f143f4b4a3dca87c09b7e4"
format_stamp: "Formatted at 2026-09-25 15:13:33 on dist-test-slave-p468"
server_key: "89b50309b9c08556a9d17055e1a6fd8d"
server_key_iv: "6fc3f259d27687abcbc004ba2da353be"
server_key_version: "encryptionkey@0"
I20260925 15:13:34.245009 18922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:13:34.256904 18922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:13:34.260114 18922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:13:34.296248 18922 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.61:35001
I20260925 15:13:34.296270 19092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.61:35001 every 8 connection(s)
I20260925 15:13:34.297183 18922 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/master-1/data/info.pb
I20260925 15:13:34.304376 19093 sys_catalog.cc:263] Verifying existing consensus state
I20260925 15:13:34.306841 19093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea8b382304f143f4b4a3dca87c09b7e4: Bootstrap starting.
I20260925 15:13:34.319053 19093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea8b382304f143f4b4a3dca87c09b7e4: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 15:13:34.319520 19093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea8b382304f143f4b4a3dca87c09b7e4: Bootstrap complete.
I20260925 15:13:34.320533 19093 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea8b382304f143f4b4a3dca87c09b7e4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c1233ca1ad3745eca6025d08fc59d7f3" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 44383 } } peers { permanent_uuid: "ea8b382304f143f4b4a3dca87c09b7e4" member_type: NON_VOTER last_known_addr { host: "127.12.8.61" port: 35001 } attrs { promote: true } }
I20260925 15:13:34.320755 19093 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea8b382304f143f4b4a3dca87c09b7e4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ea8b382304f143f4b4a3dca87c09b7e4, State: Initialized, Role: LEARNER
I20260925 15:13:34.320993 19093 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea8b382304f143f4b4a3dca87c09b7e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c1233ca1ad3745eca6025d08fc59d7f3" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 44383 } } peers { permanent_uuid: "ea8b382304f143f4b4a3dca87c09b7e4" member_type: NON_VOTER last_known_addr { host: "127.12.8.61" port: 35001 } attrs { promote: true } }
I20260925 15:13:34.321785 19096 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea8b382304f143f4b4a3dca87c09b7e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c1233ca1ad3745eca6025d08fc59d7f3" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 44383 } } peers { permanent_uuid: "ea8b382304f143f4b4a3dca87c09b7e4" member_type: NON_VOTER last_known_addr { host: "127.12.8.61" port: 35001 } attrs { promote: true } } }
I20260925 15:13:34.322088 19093 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea8b382304f143f4b4a3dca87c09b7e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:13:34.322239 19096 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea8b382304f143f4b4a3dca87c09b7e4 [sys.catalog]: This master's current role is: LEARNER
I20260925 15:13:34.329771 19107 catalog_manager.cc:1259] Loaded cluster ID: 5911822a4ce34211bb32228d45a52d8f
I20260925 15:13:34.329882 19107 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ea8b382304f143f4b4a3dca87c09b7e4: loading cluster ID for follower catalog manager: success
I20260925 15:13:34.332244 19107 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ea8b382304f143f4b4a3dca87c09b7e4: acquiring CA information for follower catalog manager: success
I20260925 15:13:34.333926 19107 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ea8b382304f143f4b4a3dca87c09b7e4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 15:13:34.515709 18917 heartbeater.cc:503] Master 127.12.8.62:44383 was elected leader, sending a full tablet report...
W20260925 15:13:34.566629 19030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:13:34.566907 19030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:13:34.567102 19030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:13:34.574316 19030 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:13:34.574414 19030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:13:34.574512 19030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:13:34.574609 19030 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:13:34.574693 19030 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:13:34.584486 19030 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:36421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.12.8.62:44383,127.12.8.61:35001,127.12.8.60:41857
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/krb5kdc/oryx_127.12.8.60.keytab
--principal=oryx/127.12.8.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.60:41857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.12.8.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:13:34.585752 19030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:13:34.587083 19030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:13:34.599108 19112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:34.599084 19115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:34.599129 19113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:13:34.603677 19030 server_base.cc:1034] running on GCE node
I20260925 15:13:34.604489 19030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:13:34.605861 19030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:13:34.607100 19030 hybrid_clock.cc:648] HybridClock initialized: now 1790349214607046 us; error 93 us; skew 500 ppm
I20260925 15:13:34.607437 19030 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:13:34 dist-test-slave-p468 krb5kdc[18415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349214, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.8.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:13:34.610718 19030 init.cc:377] Logged in from keytab as oryx/127.12.8.60@KRBTEST.COM (short username oryx)
I20260925 15:13:34.612229 19030 webserver.cc:460] Webserver started at http://127.12.8.60:45979/ using document root <none> and password file <none>
I20260925 15:13:34.612907 19030 fs_manager.cc:359] Metadata directory not provided
I20260925 15:13:34.613054 19030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:13:34.613350 19030 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:13:34.616307 19030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/master-2/data/instance:
uuid: "01c12756ade6445f92d995ab935f60d0"
format_stamp: "Formatted at 2026-09-25 15:13:34 on dist-test-slave-p468"
server_key: "2c60d6ac435e42b674745600ba2dbcfa"
server_key_iv: "b112b67bcd68ed53827085ee0fe69192"
server_key_version: "encryptionkey@0"
I20260925 15:13:34.617101 19030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/master-2/wal/instance:
uuid: "01c12756ade6445f92d995ab935f60d0"
format_stamp: "Formatted at 2026-09-25 15:13:34 on dist-test-slave-p468"
server_key: "2c60d6ac435e42b674745600ba2dbcfa"
server_key_iv: "b112b67bcd68ed53827085ee0fe69192"
server_key_version: "encryptionkey@0"
I20260925 15:13:34.622470 19030 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.006s
I20260925 15:13:34.625914 19122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:34.627545 19030 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:13:34.627727 19030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/master-2/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/master-2/wal
uuid: "01c12756ade6445f92d995ab935f60d0"
format_stamp: "Formatted at 2026-09-25 15:13:34 on dist-test-slave-p468"
server_key: "2c60d6ac435e42b674745600ba2dbcfa"
server_key_iv: "b112b67bcd68ed53827085ee0fe69192"
server_key_version: "encryptionkey@0"
I20260925 15:13:34.627913 19030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:13:34.652716 19030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:13:34.657274 19030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:13:34.657580 19030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:13:34.669894 19132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:34.661586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49482 (local address 127.12.8.61:35001)
0925 15:13:34.662475 (+ 889us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:34.662479 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:34.662495 (+ 16us) server_negotiation.cc:407] Connection header received
0925 15:13:34.662554 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:34.662557 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:34.662620 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:34.662746 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:13:34.663271 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:34.665114 (+ 1843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:34.667442 (+ 2328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:34.668670 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:34.668844 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:34.669101 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:34.669342 (+ 241us) server_negotiation.cc:299] Negotiation successful
0925 15:13:34.669510 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":780,"thread_start_us":662,"threads_started":1}
I20260925 15:13:34.669929 19130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:34.661406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.61:35001 (local address 127.0.0.1:49482)
0925 15:13:34.661825 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:34.661850 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:34.661995 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:34.662787 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:34.662793 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:34.662832 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:13:34.663067 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:34.663082 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:34.665298 (+ 2216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:34.665306 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:34.667246 (+ 1940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:34.667259 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:34.668875 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:34.668881 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:34.669109 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:34.669185 (+ 76us) client_negotiation.cc:769] Sending connection context
0925 15:13:34.669303 (+ 118us) client_negotiation.cc:240] Negotiation successful
0925 15:13:34.669502 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":223,"threads_started":1}
I20260925 15:13:34.671324 19068 raft_consensus.cc:1203] T 00000000000000000000000000000000 P ea8b382304f143f4b4a3dca87c09b7e4 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260925 15:13:34.672004 19096 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea8b382304f143f4b4a3dca87c09b7e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1233ca1ad3745eca6025d08fc59d7f3. Latest consensus state: current_term: 1 leader_uuid: "c1233ca1ad3745eca6025d08fc59d7f3" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c1233ca1ad3745eca6025d08fc59d7f3" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 44383 } } peers { permanent_uuid: "ea8b382304f143f4b4a3dca87c09b7e4" member_type: NON_VOTER last_known_addr { host: "127.12.8.61" port: 35001 } attrs { promote: true } } }
I20260925 15:13:34.672271 19096 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea8b382304f143f4b4a3dca87c09b7e4 [sys.catalog]: This master's current role is: LEARNER
I20260925 15:13:34.677858 19030 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.60:41857
I20260925 15:13:34.677901 19176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.60:41857 every 8 connection(s)
I20260925 15:13:34.679334 19030 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/master-2/data/info.pb
I20260925 15:13:34.683027 19177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:13:34.687750 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 19030
I20260925 15:13:34.689985 19177 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.8.62" port: 44383 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:13:34 dist-test-slave-p468 krb5kdc[18415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349214, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.8.60@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM
Sep 25 15:13:34 dist-test-slave-p468 krb5kdc[18415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349211, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.8.60@KRBTEST.COM
I20260925 15:13:34.707475 18652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:34.692764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60668 (local address 127.12.8.62:44383)
0925 15:13:34.692953 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:34.692959 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:34.693333 (+ 374us) server_negotiation.cc:407] Connection header received
0925 15:13:34.693495 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:34.693501 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:34.693583 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:34.693699 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:34.695000 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:34.695715 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:34.697330 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:34.697806 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:34.697893 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:34.701621 (+ 3728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:34.701627 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:34.701631 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:34.701668 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:34.704427 (+ 2759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:34.705267 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:34.705274 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:34.705278 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:34.705404 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:34.705805 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:34.705813 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:34.705817 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:34.706072 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:34.706259 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:34.706912 (+ 653us) server_negotiation.cc:299] Negotiation successful
0925 15:13:34.707123 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 15:13:34.708498 19180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:34.689982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51080 (local address 127.12.8.60:41857)
0925 15:13:34.690792 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:34.690803 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:34.690844 (+ 41us) server_negotiation.cc:407] Connection header received
0925 15:13:34.692595 (+ 1751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:34.692621 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:34.693168 (+ 547us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:34.693508 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:34.694909 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:34.696248 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:34.697495 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:34.698079 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:34.698336 (+ 257us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:34.702479 (+ 4143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:34.702487 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:34.702498 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:34.702527 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:34.705119 (+ 2592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:34.705741 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:34.705748 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:34.705750 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:34.705863 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:34.706266 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:34.706273 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:34.706276 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:34.706648 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:34.706943 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:34.707241 (+ 298us) server_negotiation.cc:299] Negotiation successful
0925 15:13:34.707469 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":65,"server-negotiator.queue_time_us":615,"spinlock_wait_cycles":1408,"thread_start_us":384,"threads_started":1}
I20260925 15:13:34.708588 19181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:34.692528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:44383 (local address 127.0.0.1:60668)
0925 15:13:34.693178 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:34.693226 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:34.693381 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:34.693767 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:34.693773 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:34.694447 (+ 674us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:13:34.694790 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:34.694807 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:34.695868 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:34.695874 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:34.697191 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:34.697202 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:34.698007 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:34.698014 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:34.698276 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:34.699283 (+ 1007us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:13:34.699303 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:13:34.701423 (+ 2120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:13:34.704561 (+ 3138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:13:34.704573 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:13:34.704583 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:13:34.705152 (+ 569us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:13:34.705510 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:13:34.705523 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:13:34.705527 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:13:34.705699 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:13:34.706242 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:13:34.706250 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:13:34.706576 (+ 326us) client_negotiation.cc:769] Sending connection context
0925 15:13:34.706838 (+ 262us) client_negotiation.cc:240] Negotiation successful
0925 15:13:34.707294 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"mutex_wait_us":89,"spinlock_wait_cycles":15616,"thread_start_us":282,"threads_started":1}
I20260925 15:13:34.710296 19177 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.8.61" port: 35001 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:13:34 dist-test-slave-p468 krb5kdc[18415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349214, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.8.60@KRBTEST.COM for oryx/127.12.8.61@KRBTEST.COM
I20260925 15:13:34.722504 19132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:34.711168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49498 (local address 127.12.8.61:35001)
0925 15:13:34.711376 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:34.711381 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:34.711400 (+ 19us) server_negotiation.cc:407] Connection header received
0925 15:13:34.711566 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:34.711577 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:34.711637 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:34.711716 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:34.712301 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:34.712995 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:34.714114 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:34.714775 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:34.714885 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:34.717501 (+ 2616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:34.717506 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:34.717509 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:34.717549 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:34.720261 (+ 2712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:34.720899 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:34.720905 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:34.720911 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:34.721054 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:34.721356 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:34.721361 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:34.721364 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:34.721609 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:34.721716 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:34.722054 (+ 338us) server_negotiation.cc:299] Negotiation successful
0925 15:13:34.722286 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 15:13:34.722555 19181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:34.711081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.61:35001 (local address 127.0.0.1:49498)
0925 15:13:34.711336 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:34.711354 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:34.711440 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:34.711750 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:34.711753 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:34.712032 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:13:34.712164 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:34.712171 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:34.713109 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:34.713112 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:34.714001 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:34.714010 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:34.714897 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:34.714900 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:34.715036 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:34.715512 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:13:34.715528 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:13:34.717340 (+ 1812us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:13:34.720372 (+ 3032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:13:34.720375 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:13:34.720378 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:13:34.720747 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:13:34.721155 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:13:34.721158 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:13:34.721160 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:13:34.721271 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:13:34.721749 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:13:34.721752 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:13:34.721869 (+ 117us) client_negotiation.cc:769] Sending connection context
0925 15:13:34.721974 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 15:13:34.722270 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155}
I20260925 15:13:34.723575 19177 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.8.60" port: 41857 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:13:34 dist-test-slave-p468 krb5kdc[18415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349214, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.8.60@KRBTEST.COM for oryx/127.12.8.60@KRBTEST.COM
I20260925 15:13:34.734628 19181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:34.724127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.60:41857 (local address 127.0.0.1:51088)
0925 15:13:34.724271 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:34.724308 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:34.724390 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:34.724639 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:34.724642 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:34.724939 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:13:34.725086 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:34.725094 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:34.725996 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:34.725999 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:34.726851 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:34.726861 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:34.727445 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:34.727448 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:34.727596 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:34.728055 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:13:34.728069 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:13:34.729806 (+ 1737us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:13:34.732687 (+ 2881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:13:34.732690 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:13:34.732692 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:13:34.733000 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:13:34.733289 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:13:34.733292 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:13:34.733294 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:13:34.733423 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:13:34.733836 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:13:34.733839 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:13:34.733934 (+ 95us) client_negotiation.cc:769] Sending connection context
0925 15:13:34.734037 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 15:13:34.734285 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260925 15:13:34.734808 19180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:34.724206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51088 (local address 127.12.8.60:41857)
0925 15:13:34.724358 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:34.724361 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:34.724375 (+ 14us) server_negotiation.cc:407] Connection header received
0925 15:13:34.724470 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:34.724473 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:34.724532 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:34.724600 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:34.725191 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:34.725895 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:34.726954 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:34.727332 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:34.727425 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:34.729958 (+ 2533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:34.729962 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:34.729966 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:34.729992 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:34.732575 (+ 2583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:34.733097 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:34.733100 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:34.733102 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:34.733204 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:34.733503 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:34.733506 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:34.733508 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:34.733745 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:34.733845 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:34.734123 (+ 278us) server_negotiation.cc:299] Negotiation successful
0925 15:13:34.734431 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":102,"server-negotiator.queue_time_us":46}
I20260925 15:13:34.739164 19177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01c12756ade6445f92d995ab935f60d0: Bootstrap starting.
I20260925 15:13:34.742412 19177 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 01c12756ade6445f92d995ab935f60d0: Neither blocks nor log segments found. Creating new log.
I20260925 15:13:34.743371 19177 log.cc:824] T 00000000000000000000000000000000 P 01c12756ade6445f92d995ab935f60d0: Log is configured to *not* fsync() on all Append() calls
I20260925 15:13:34.745788 19177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01c12756ade6445f92d995ab935f60d0: No bootstrap required, opened a new log
I20260925 15:13:34.752638 19177 raft_consensus.cc:348] T 00000000000000000000000000000000 P 01c12756ade6445f92d995ab935f60d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1233ca1ad3745eca6025d08fc59d7f3" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 44383 } } peers { permanent_uuid: "ea8b382304f143f4b4a3dca87c09b7e4" member_type: VOTER last_known_addr { host: "127.12.8.61" port: 35001 } } peers { permanent_uuid: "01c12756ade6445f92d995ab935f60d0" member_type: VOTER last_known_addr { host: "127.12.8.60" port: 41857 } }
I20260925 15:13:34.752934 19177 raft_consensus.cc:374] T 00000000000000000000000000000000 P 01c12756ade6445f92d995ab935f60d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:13:34.753060 19177 raft_consensus.cc:729] T 00000000000000000000000000000000 P 01c12756ade6445f92d995ab935f60d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01c12756ade6445f92d995ab935f60d0, State: Initialized, Role: FOLLOWER
I20260925 15:13:34.753611 19177 consensus_queue.cc:260] T 00000000000000000000000000000000 P 01c12756ade6445f92d995ab935f60d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1233ca1ad3745eca6025d08fc59d7f3" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 44383 } } peers { permanent_uuid: "ea8b382304f143f4b4a3dca87c09b7e4" member_type: VOTER last_known_addr { host: "127.12.8.61" port: 35001 } } peers { permanent_uuid: "01c12756ade6445f92d995ab935f60d0" member_type: VOTER last_known_addr { host: "127.12.8.60" port: 41857 } }
I20260925 15:13:34.754520 19183 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01c12756ade6445f92d995ab935f60d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1233ca1ad3745eca6025d08fc59d7f3" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 44383 } } peers { permanent_uuid: "ea8b382304f143f4b4a3dca87c09b7e4" member_type: VOTER last_known_addr { host: "127.12.8.61" port: 35001 } } peers { permanent_uuid: "01c12756ade6445f92d995ab935f60d0" member_type: VOTER last_known_addr { host: "127.12.8.60" port: 41857 } } }
I20260925 15:13:34.754875 19183 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01c12756ade6445f92d995ab935f60d0 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:13:34.755511 19177 sys_catalog.cc:564] T 00000000000000000000000000000000 P 01c12756ade6445f92d995ab935f60d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:13:34.763583 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 18522
W20260925 15:13:34.764238 19194 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 01c12756ade6445f92d995ab935f60d0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:13:34.764361 19194 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:13:34.768983 19030 master_runner.cc:428] Detected that this master 01c12756ade6445f92d995ab935f60d0 is joining an existing cluster
I20260925 15:13:34.769095 19030 master_runner.cc:432] Initiating AddMaster RPC to add 127.12.8.60:41857
Sep 25 15:13:34 dist-test-slave-p468 krb5kdc[18415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349211, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.8.1@KRBTEST.COM
I20260925 15:13:34.786123 19196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:34.764769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49274 (local address 127.12.8.1:46055)
0925 15:13:34.765674 (+ 905us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:34.765682 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:34.765712 (+ 30us) server_negotiation.cc:407] Connection header received
0925 15:13:34.765762 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:34.765790 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:34.766020 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:34.766859 (+ 839us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:34.768393 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:34.769605 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:34.771427 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:34.771954 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:34.772091 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:34.778756 (+ 6665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:34.778775 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:34.778794 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:34.778859 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:34.782632 (+ 3773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:34.783486 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:34.783498 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:34.783503 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:34.783655 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:34.784152 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:34.784156 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:34.784159 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:34.784581 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:34.784849 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:34.785449 (+ 600us) server_negotiation.cc:299] Negotiation successful
0925 15:13:34.785693 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":719,"thread_start_us":429,"threads_started":1}
I20260925 15:13:34.788412 18645 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:13:34.809188 19180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:34.782999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51096 (local address 127.12.8.60:41857)
0925 15:13:34.783167 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:34.783172 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:34.794002 (+ 10830us) server_negotiation.cc:407] Connection header received
0925 15:13:34.794355 (+ 353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:34.794360 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:34.794420 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:34.794480 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:34.795963 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:34.797135 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:34.799091 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:34.799840 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:34.799931 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:34.802267 (+ 2336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:34.802273 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:34.802278 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:34.802309 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:34.805846 (+ 3537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:34.806653 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:34.806658 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:34.806661 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:34.806794 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:34.807238 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:34.807243 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:34.807247 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:34.807595 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:34.807722 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:34.808218 (+ 496us) server_negotiation.cc:299] Negotiation successful
0925 15:13:34.808431 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 15:13:34.810149 19203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:34.783269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.60:41857 (local address 127.0.0.1:51096)
0925 15:13:34.793913 (+ 10644us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:34.793931 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:34.794226 (+ 295us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:34.794671 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:34.794677 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:34.795413 (+ 736us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:13:34.795778 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:34.795790 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:34.797345 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:34.797351 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:34.798946 (+ 1595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:34.798960 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:34.800200 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:34.800207 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:34.800364 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:34.801030 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:13:34.801052 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:13:34.802064 (+ 1012us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:13:34.805993 (+ 3929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:13:34.805998 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:13:34.806002 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:13:34.806529 (+ 527us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:13:34.806935 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:13:34.806941 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:13:34.806947 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:13:34.807112 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:13:34.807785 (+ 673us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:13:34.807790 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:13:34.807955 (+ 165us) client_negotiation.cc:769] Sending connection context
0925 15:13:34.809432 (+ 1477us) client_negotiation.cc:240] Negotiation successful
0925 15:13:34.809699 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10512,"thread_start_us":10753,"threads_started":1}
I20260925 15:13:34.812281 18652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:34.773302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60672 (local address 127.12.8.62:44383)
0925 15:13:34.773475 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:34.773482 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:34.782818 (+ 9336us) server_negotiation.cc:407] Connection header received
0925 15:13:34.783070 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:34.783075 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:34.783156 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:34.783226 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:34.784729 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:34.785810 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:34.787741 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:34.788319 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:34.788389 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:34.799664 (+ 11275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:34.799671 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:34.799675 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:34.799720 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:34.803556 (+ 3836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:34.808456 (+ 4900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:34.808462 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:34.808466 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:34.808631 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:34.809707 (+ 1076us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:34.809712 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:34.809716 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:34.810031 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:34.810174 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:34.811718 (+ 1544us) server_negotiation.cc:299] Negotiation successful
0925 15:13:34.811932 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 15:13:34.813452 19201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:34.773793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:44383 (local address 127.0.0.1:60672)
0925 15:13:34.782732 (+ 8939us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:34.782749 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:34.782917 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:34.783442 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:34.783450 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:34.784163 (+ 713us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:13:34.784558 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:34.784571 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:34.785977 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:34.785982 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:34.787589 (+ 1607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:34.787602 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:34.788664 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:34.788672 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:34.788898 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:34.789664 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:13:34.789692 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:13:34.799375 (+ 9683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:13:34.803724 (+ 4349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:13:34.803729 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:13:34.803734 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:13:34.804180 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:13:34.808772 (+ 4592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:13:34.808777 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:13:34.808781 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:13:34.808952 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:13:34.810275 (+ 1323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:13:34.810282 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:13:34.810481 (+ 199us) client_negotiation.cc:769] Sending connection context
0925 15:13:34.812735 (+ 2254us) client_negotiation.cc:240] Negotiation successful
0925 15:13:34.813030 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8797,"thread_start_us":6928,"threads_started":1}
I20260925 15:13:34.814571 19132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:34.773905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49504 (local address 127.12.8.61:35001)
0925 15:13:34.774108 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:34.774117 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:34.780605 (+ 6488us) server_negotiation.cc:407] Connection header received
0925 15:13:34.780793 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:34.780801 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:34.780907 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:34.781036 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:34.790658 (+ 9622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:34.791391 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:34.793573 (+ 2182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:34.794286 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:34.794381 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:34.804365 (+ 9984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:34.804371 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:34.804376 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:34.804417 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:34.807715 (+ 3298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:34.809085 (+ 1370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:34.809090 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:34.809094 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:34.809235 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:34.810681 (+ 1446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:34.810687 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:34.810690 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:34.811020 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:34.811146 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:34.813993 (+ 2847us) server_negotiation.cc:299] Negotiation successful
0925 15:13:34.814234 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 15:13:34.815526 19202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:34.773369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.61:35001 (local address 127.0.0.1:49504)
0925 15:13:34.780487 (+ 7118us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:34.780507 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:34.780654 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:34.789109 (+ 8455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:34.789117 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:34.789955 (+ 838us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:13:34.790412 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:34.790428 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:34.791632 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:34.791640 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:34.793398 (+ 1758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:34.793412 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:34.794675 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:34.794682 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:34.794894 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:34.795639 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:13:34.795663 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:13:34.798978 (+ 3315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:13:34.807876 (+ 8898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:13:34.807881 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:13:34.807885 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:13:34.808316 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:13:34.809400 (+ 1084us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:13:34.809403 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:13:34.809406 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:13:34.809585 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:13:34.811220 (+ 1635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:13:34.811225 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:13:34.811437 (+ 212us) client_negotiation.cc:769] Sending connection context
0925 15:13:34.814787 (+ 3350us) client_negotiation.cc:240] Negotiation successful
0925 15:13:34.815112 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6962,"thread_start_us":9032,"threads_started":1}
Sep 25 15:13:34 dist-test-slave-p468 krb5kdc[18415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349211, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.8.62@KRBTEST.COM for oryx/127.12.8.60@KRBTEST.COM
I20260925 15:13:34.835031 19130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:34.818777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.60:41857 (local address 127.0.0.1:51108)
0925 15:13:34.818945 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:34.818963 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:34.819085 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:34.819653 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:34.819660 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:34.820126 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:13:34.820304 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:34.820315 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:34.821772 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:34.821778 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:34.823195 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:34.823210 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:34.824292 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:34.824300 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:34.824471 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:34.825067 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:13:34.825093 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:13:34.828011 (+ 2918us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:13:34.832172 (+ 4161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:13:34.832178 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:13:34.832183 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:13:34.832753 (+ 570us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:13:34.833146 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:13:34.833152 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:13:34.833156 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:13:34.833311 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:13:34.833946 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:13:34.833951 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:13:34.834147 (+ 196us) client_negotiation.cc:769] Sending connection context
0925 15:13:34.834340 (+ 193us) client_negotiation.cc:240] Negotiation successful
0925 15:13:34.834627 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260925 15:13:34.835039 19180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:34.818845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51108 (local address 127.12.8.60:41857)
0925 15:13:34.818988 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:34.818995 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:34.819013 (+ 18us) server_negotiation.cc:407] Connection header received
0925 15:13:34.819257 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:34.819262 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:34.819364 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:34.819453 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:34.820472 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:34.821604 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:34.823406 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:34.824142 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:34.824266 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:34.828241 (+ 3975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:34.828248 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:34.828253 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:34.828289 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:34.831957 (+ 3668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:34.832873 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:34.832881 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:34.832889 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:34.833025 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:34.833426 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:34.833431 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:34.833435 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:34.833778 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:34.833921 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:34.834441 (+ 520us) server_negotiation.cc:299] Negotiation successful
0925 15:13:34.834679 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 15:13:34.836809 18460 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.12.8.60:41857: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "01c12756ade6445f92d995ab935f60d0"
member_type: NON_VOTER
last_known_addr {
host: "127.12.8.60"
port: 41857
}
attrs {
promote: true
}
}
dest_uuid: "c1233ca1ad3745eca6025d08fc59d7f3"
cas_config_opid_index: 5
I20260925 15:13:34.837842 18460 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1233ca1ad3745eca6025d08fc59d7f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c1233ca1ad3745eca6025d08fc59d7f3" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 44383 } } peers { permanent_uuid: "ea8b382304f143f4b4a3dca87c09b7e4" member_type: NON_VOTER last_known_addr { host: "127.12.8.61" port: 35001 } attrs { promote: true } } peers { permanent_uuid: "01c12756ade6445f92d995ab935f60d0" member_type: NON_VOTER last_known_addr { host: "127.12.8.60" port: 41857 } attrs { promote: true } }
I20260925 15:13:34.839591 19129 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c1233ca1ad3745eca6025d08fc59d7f3 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 01c12756ade6445f92d995ab935f60d0 (127.12.8.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c1233ca1ad3745eca6025d08fc59d7f3" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 44383 } } peers { permanent_uuid: "ea8b382304f143f4b4a3dca87c09b7e4" member_type: NON_VOTER last_known_addr { host: "127.12.8.61" port: 35001 } attrs { promote: true } } peers { permanent_uuid: "01c12756ade6445f92d995ab935f60d0" member_type: NON_VOTER last_known_addr { host: "127.12.8.60" port: 41857 } attrs { promote: true } } }
I20260925 15:13:34.840329 19068 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ea8b382304f143f4b4a3dca87c09b7e4 [term 1 LEARNER]: Refusing update from remote peer c1233ca1ad3745eca6025d08fc59d7f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 15:13:34.841686 19152 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 01c12756ade6445f92d995ab935f60d0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:13:34.844517 19129 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.12.8.60:41857
I20260925 15:13:34.849311 19205 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c1233ca1ad3745eca6025d08fc59d7f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea8b382304f143f4b4a3dca87c09b7e4" member_type: NON_VOTER last_known_addr { host: "127.12.8.61" port: 35001 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260925 15:13:34.849915 19206 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1233ca1ad3745eca6025d08fc59d7f3 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c1233ca1ad3745eca6025d08fc59d7f3" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c1233ca1ad3745eca6025d08fc59d7f3" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 44383 } } peers { permanent_uuid: "ea8b382304f143f4b4a3dca87c09b7e4" member_type: NON_VOTER last_known_addr { host: "127.12.8.61" port: 35001 } attrs { promote: true } } peers { permanent_uuid: "01c12756ade6445f92d995ab935f60d0" member_type: NON_VOTER last_known_addr { host: "127.12.8.60" port: 41857 } attrs { promote: true } } }
I20260925 15:13:34.850332 19206 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1233ca1ad3745eca6025d08fc59d7f3 [sys.catalog]: This master's current role is: LEADER
I20260925 15:13:34.858728 19068 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ea8b382304f143f4b4a3dca87c09b7e4 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 01c12756ade6445f92d995ab935f60d0 (127.12.8.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c1233ca1ad3745eca6025d08fc59d7f3" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 44383 } } peers { permanent_uuid: "ea8b382304f143f4b4a3dca87c09b7e4" member_type: NON_VOTER last_known_addr { host: "127.12.8.61" port: 35001 } attrs { promote: true } } peers { permanent_uuid: "01c12756ade6445f92d995ab935f60d0" member_type: NON_VOTER last_known_addr { host: "127.12.8.60" port: 41857 } attrs { promote: true } } }
I20260925 15:13:34.862540 19030 master.cc:537] Master@127.12.8.60:41857 shutting down...
I20260925 15:13:34.862797 19152 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 01c12756ade6445f92d995ab935f60d0 [term 1 FOLLOWER]: Refusing update from remote peer c1233ca1ad3745eca6025d08fc59d7f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 15:13:34.863195 19096 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea8b382304f143f4b4a3dca87c09b7e4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c1233ca1ad3745eca6025d08fc59d7f3" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c1233ca1ad3745eca6025d08fc59d7f3" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 44383 } } peers { permanent_uuid: "ea8b382304f143f4b4a3dca87c09b7e4" member_type: NON_VOTER last_known_addr { host: "127.12.8.61" port: 35001 } attrs { promote: true } } peers { permanent_uuid: "01c12756ade6445f92d995ab935f60d0" member_type: NON_VOTER last_known_addr { host: "127.12.8.60" port: 41857 } attrs { promote: true } } }
I20260925 15:13:34.863448 19096 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea8b382304f143f4b4a3dca87c09b7e4 [sys.catalog]: This master's current role is: LEARNER
I20260925 15:13:34.876797 19205 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c1233ca1ad3745eca6025d08fc59d7f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01c12756ade6445f92d995ab935f60d0" member_type: NON_VOTER last_known_addr { host: "127.12.8.60" port: 41857 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:13:34.878379 19030 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 01c12756ade6445f92d995ab935f60d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:13:34.878700 19129 raft_consensus.cc:1050] T 00000000000000000000000000000000 P c1233ca1ad3745eca6025d08fc59d7f3: attempting to promote NON_VOTER ea8b382304f143f4b4a3dca87c09b7e4 to VOTER
I20260925 15:13:34.878804 19030 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 01c12756ade6445f92d995ab935f60d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:13:34.879009 19030 tablet_replica.cc:330] T 00000000000000000000000000000000 P 01c12756ade6445f92d995ab935f60d0: stopping tablet replica
I20260925 15:13:34.885344 19130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:34.884811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.60:41857 (local address 127.0.0.1:51120)
0925 15:13:34.885015 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:34.885147 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.60:41857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260925 15:13:34.885721 19030 master.cc:552] Master@127.12.8.60:41857 shutdown complete.
I20260925 15:13:34.885802 19030 master_runner.cc:305] Clearing existing system tablet
I20260925 15:13:34.885634 19129 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1233ca1ad3745eca6025d08fc59d7f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c1233ca1ad3745eca6025d08fc59d7f3" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 44383 } } peers { permanent_uuid: "ea8b382304f143f4b4a3dca87c09b7e4" member_type: VOTER last_known_addr { host: "127.12.8.61" port: 35001 } attrs { promote: false } } peers { permanent_uuid: "01c12756ade6445f92d995ab935f60d0" member_type: NON_VOTER last_known_addr { host: "127.12.8.60" port: 41857 } attrs { promote: true } }
I20260925 15:13:34.886178 19030 fs_manager.cc:359] Metadata directory not provided
I20260925 15:13:34.886329 19030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:13:34.888006 19130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:34.887609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.60:41857 (local address 127.0.0.1:51130)
0925 15:13:34.887756 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:34.887840 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.60:41857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260925 15:13:34.889333 19068 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ea8b382304f143f4b4a3dca87c09b7e4 [term 1 LEARNER]: Refusing update from remote peer c1233ca1ad3745eca6025d08fc59d7f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260925 15:13:34.890129 19205 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c1233ca1ad3745eca6025d08fc59d7f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea8b382304f143f4b4a3dca87c09b7e4" member_type: VOTER last_known_addr { host: "127.12.8.61" port: 35001 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260925 15:13:34.893463 19130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:34.892980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.60:41857 (local address 127.0.0.1:51134)
0925 15:13:34.893154 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:34.893256 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.60:41857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260925 15:13:34.893782 18444 consensus_peers.cc:487] T 00000000000000000000000000000000 P c1233ca1ad3745eca6025d08fc59d7f3 -> Peer 01c12756ade6445f92d995ab935f60d0 (127.12.8.60:41857): Couldn't send request to peer 01c12756ade6445f92d995ab935f60d0. Status: Network error: Client connection negotiation failed: client connection to 127.12.8.60:41857: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 15:13:34.894364 19206 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c1233ca1ad3745eca6025d08fc59d7f3 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, ea8b382304f143f4b4a3dca87c09b7e4 (127.12.8.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c1233ca1ad3745eca6025d08fc59d7f3" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 44383 } } peers { permanent_uuid: "ea8b382304f143f4b4a3dca87c09b7e4" member_type: VOTER last_known_addr { host: "127.12.8.61" port: 35001 } attrs { promote: false } } peers { permanent_uuid: "01c12756ade6445f92d995ab935f60d0" member_type: NON_VOTER last_known_addr { host: "127.12.8.60" port: 41857 } attrs { promote: true } } }
I20260925 15:13:34.901163 19068 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ea8b382304f143f4b4a3dca87c09b7e4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, ea8b382304f143f4b4a3dca87c09b7e4 (127.12.8.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c1233ca1ad3745eca6025d08fc59d7f3" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 44383 } } peers { permanent_uuid: "ea8b382304f143f4b4a3dca87c09b7e4" member_type: VOTER last_known_addr { host: "127.12.8.61" port: 35001 } attrs { promote: false } } peers { permanent_uuid: "01c12756ade6445f92d995ab935f60d0" member_type: NON_VOTER last_known_addr { host: "127.12.8.60" port: 41857 } attrs { promote: true } } }
I20260925 15:13:34.906476 19096 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea8b382304f143f4b4a3dca87c09b7e4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c1233ca1ad3745eca6025d08fc59d7f3" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c1233ca1ad3745eca6025d08fc59d7f3" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 44383 } } peers { permanent_uuid: "ea8b382304f143f4b4a3dca87c09b7e4" member_type: VOTER last_known_addr { host: "127.12.8.61" port: 35001 } attrs { promote: false } } peers { permanent_uuid: "01c12756ade6445f92d995ab935f60d0" member_type: NON_VOTER last_known_addr { host: "127.12.8.60" port: 41857 } attrs { promote: true } } }
I20260925 15:13:34.906821 19096 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea8b382304f143f4b4a3dca87c09b7e4 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:13:34.912159 19205 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1233ca1ad3745eca6025d08fc59d7f3 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c1233ca1ad3745eca6025d08fc59d7f3" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c1233ca1ad3745eca6025d08fc59d7f3" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 44383 } } peers { permanent_uuid: "ea8b382304f143f4b4a3dca87c09b7e4" member_type: VOTER last_known_addr { host: "127.12.8.61" port: 35001 } attrs { promote: false } } peers { permanent_uuid: "01c12756ade6445f92d995ab935f60d0" member_type: NON_VOTER last_known_addr { host: "127.12.8.60" port: 41857 } attrs { promote: true } } }
I20260925 15:13:34.912525 19205 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1233ca1ad3745eca6025d08fc59d7f3 [sys.catalog]: This master's current role is: LEADER
I20260925 15:13:34.912974 19206 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1233ca1ad3745eca6025d08fc59d7f3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c1233ca1ad3745eca6025d08fc59d7f3" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c1233ca1ad3745eca6025d08fc59d7f3" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 44383 } } peers { permanent_uuid: "ea8b382304f143f4b4a3dca87c09b7e4" member_type: VOTER last_known_addr { host: "127.12.8.61" port: 35001 } attrs { promote: false } } peers { permanent_uuid: "01c12756ade6445f92d995ab935f60d0" member_type: NON_VOTER last_known_addr { host: "127.12.8.60" port: 41857 } attrs { promote: true } } }
I20260925 15:13:34.913275 19206 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1233ca1ad3745eca6025d08fc59d7f3 [sys.catalog]: This master's current role is: LEADER
I20260925 15:13:34.914042 19030 fs_manager.cc:709] Time spent opening directory manager: real 0.027s user 0.010s sys 0.017s
I20260925 15:13:34.915550 19213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:34.915898 19030 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:13:34.916174 19030 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:13:34.916275 19030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/master-2/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/master-2/wal
uuid: "01c12756ade6445f92d995ab935f60d0"
format_stamp: "Formatted at 2026-09-25 15:13:34 on dist-test-slave-p468"
server_key: "2c60d6ac435e42b674745600ba2dbcfa"
server_key_iv: "b112b67bcd68ed53827085ee0fe69192"
server_key_version: "encryptionkey@0"
I20260925 15:13:34.919642 19030 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 01c12756ade6445f92d995ab935f60d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:13:34.925424 19030 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 01c12756ade6445f92d995ab935f60d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 15:13:34.925606 19030 log.cc:1196] T 00000000000000000000000000000000 P 01c12756ade6445f92d995ab935f60d0: Deleting WAL directory at /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260925 15:13:34.926052 19030 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 01c12756ade6445f92d995ab935f60d0: Deleting consensus metadata
I20260925 15:13:34.926079 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 18522
I20260925 15:13:34.926328 19030 master_runner.cc:315] Copying system tablet from 127.12.8.62:44383
I20260925 15:13:34.928714 19030 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 01c12756ade6445f92d995ab935f60d0: tablet copy: Beginning tablet copy session from remote peer at address 127.12.8.62:44383
I20260925 15:13:34.948436 18652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:34.929466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60684 (local address 127.12.8.62:44383)
0925 15:13:34.929722 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:34.929732 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:34.935061 (+ 5329us) server_negotiation.cc:407] Connection header received
0925 15:13:34.935266 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:34.935272 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:34.935345 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:34.935454 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:34.936279 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:34.936996 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:34.938583 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:34.939232 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:34.939323 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:34.941943 (+ 2620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:34.941949 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:34.941954 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:34.941992 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:34.945552 (+ 3560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:34.946307 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:34.946313 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:34.946317 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:34.946459 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:34.946854 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:34.946858 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:34.946861 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:34.947199 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:34.947335 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:34.947857 (+ 522us) server_negotiation.cc:299] Negotiation successful
0925 15:13:34.948076 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260925 15:13:34.949419 19219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:34.929436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:44383 (local address 127.0.0.1:60684)
0925 15:13:34.934951 (+ 5515us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:34.934966 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:34.935128 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:34.935476 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:34.935480 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:34.935932 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:13:34.936123 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:34.936132 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:34.937131 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:34.937137 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:34.938412 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:34.938426 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:34.939611 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:34.939616 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:34.939787 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:34.940540 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:13:34.940563 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:13:34.941691 (+ 1128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:13:34.945695 (+ 4004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:13:34.945701 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:13:34.945705 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:13:34.946158 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:13:34.946592 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:13:34.946597 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:13:34.946601 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:13:34.946746 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:13:34.947417 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:13:34.947422 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:13:34.947594 (+ 172us) client_negotiation.cc:769] Sending connection context
0925 15:13:34.948677 (+ 1083us) client_negotiation.cc:240] Negotiation successful
0925 15:13:34.948985 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5360,"thread_start_us":5222,"threads_started":1}
I20260925 15:13:34.950559 18480 tablet_copy_service.cc:140] P c1233ca1ad3745eca6025d08fc59d7f3: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 01c12756ade6445f92d995ab935f60d0 ({username='oryx', principal='oryx/127.12.8.60@KRBTEST.COM'} at 127.0.0.1:60684)
I20260925 15:13:34.950608 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.1:46055
--local_ip_for_outbound_sockets=127.12.8.1
--tserver_master_addrs=127.12.8.62:44383
--webserver_port=37297
--webserver_interface=127.12.8.1
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/krb5kdc/oryx_127.12.8.1.keytab
--principal=oryx/127.12.8.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:36421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.12.8.62:44383,127.12.8.61:35001
--tserver_master_addrs=127.12.8.62:44383,127.12.8.61:35001,127.12.8.60:41857 with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:13:34.950726 18480 tablet_copy_service.cc:161] P c1233ca1ad3745eca6025d08fc59d7f3: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 01c12756ade6445f92d995ab935f60d0 at {username='oryx', principal='oryx/127.12.8.60@KRBTEST.COM'} at 127.0.0.1:60684: session id = 01c12756ade6445f92d995ab935f60d0-00000000000000000000000000000000
I20260925 15:13:34.954573 18480 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P c1233ca1ad3745eca6025d08fc59d7f3: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 15:13:34.956476 19030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 15:13:34.963380 19030 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 01c12756ade6445f92d995ab935f60d0: tablet copy: Starting download of 0 data blocks...
I20260925 15:13:34.963654 19030 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 01c12756ade6445f92d995ab935f60d0: tablet copy: Starting download of 1 WAL segments...
I20260925 15:13:34.965997 19030 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 01c12756ade6445f92d995ab935f60d0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 15:13:34.970486 18480 tablet_copy_service.cc:342] P c1233ca1ad3745eca6025d08fc59d7f3: Request end of tablet copy session 01c12756ade6445f92d995ab935f60d0-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.12.8.60@KRBTEST.COM'} at 127.0.0.1:60684
I20260925 15:13:34.970624 18480 tablet_copy_service.cc:433] P c1233ca1ad3745eca6025d08fc59d7f3: ending tablet copy session 01c12756ade6445f92d995ab935f60d0-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 01c12756ade6445f92d995ab935f60d0
I20260925 15:13:34.983755 19030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:13:34.986483 19221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:34.991958 19222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:34.999207 19224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:13:35.011797 19030 server_base.cc:1034] running on GCE node
I20260925 15:13:35.012156 19030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:13:35.012871 19030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:13:35.014078 19030 hybrid_clock.cc:648] HybridClock initialized: now 1790349215014038 us; error 33 us; skew 500 ppm
I20260925 15:13:35.014441 19030 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:13:35 dist-test-slave-p468 krb5kdc[18415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349215, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.8.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:13:35.016644 19030 init.cc:377] Logged in from keytab as oryx/127.12.8.60@KRBTEST.COM (short username oryx)
I20260925 15:13:35.019510 19030 webserver.cc:460] Webserver started at http://127.12.8.60:40853/ using document root <none> and password file <none>
I20260925 15:13:35.019856 19030 fs_manager.cc:359] Metadata directory not provided
I20260925 15:13:35.019948 19030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:13:35.023175 19030 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260925 15:13:35.025089 19231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:35.025543 19030 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260925 15:13:35.025689 19030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/master-2/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/master-2/wal
uuid: "01c12756ade6445f92d995ab935f60d0"
format_stamp: "Formatted at 2026-09-25 15:13:34 on dist-test-slave-p468"
server_key: "2c60d6ac435e42b674745600ba2dbcfa"
server_key_iv: "b112b67bcd68ed53827085ee0fe69192"
server_key_version: "encryptionkey@0"
I20260925 15:13:35.025928 19030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:13:35.063937 19030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:13:35.068060 19030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:13:35.091190 19030 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.60:41857
I20260925 15:13:35.091234 19282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.60:41857 every 8 connection(s)
I20260925 15:13:35.092247 19030 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/master-2/data/info.pb
I20260925 15:13:35.097681 19283 sys_catalog.cc:263] Verifying existing consensus state
I20260925 15:13:35.101832 19283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01c12756ade6445f92d995ab935f60d0: Bootstrap starting.
I20260925 15:13:35.120348 19283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01c12756ade6445f92d995ab935f60d0: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 15:13:35.120891 19283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01c12756ade6445f92d995ab935f60d0: Bootstrap complete.
I20260925 15:13:35.122200 19283 raft_consensus.cc:348] T 00000000000000000000000000000000 P 01c12756ade6445f92d995ab935f60d0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c1233ca1ad3745eca6025d08fc59d7f3" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 44383 } } peers { permanent_uuid: "ea8b382304f143f4b4a3dca87c09b7e4" member_type: VOTER last_known_addr { host: "127.12.8.61" port: 35001 } attrs { promote: false } } peers { permanent_uuid: "01c12756ade6445f92d995ab935f60d0" member_type: NON_VOTER last_known_addr { host: "127.12.8.60" port: 41857 } attrs { promote: true } }
I20260925 15:13:35.122486 19283 raft_consensus.cc:729] T 00000000000000000000000000000000 P 01c12756ade6445f92d995ab935f60d0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 01c12756ade6445f92d995ab935f60d0, State: Initialized, Role: LEARNER
I20260925 15:13:35.122766 19283 consensus_queue.cc:260] T 00000000000000000000000000000000 P 01c12756ade6445f92d995ab935f60d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c1233ca1ad3745eca6025d08fc59d7f3" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 44383 } } peers { permanent_uuid: "ea8b382304f143f4b4a3dca87c09b7e4" member_type: VOTER last_known_addr { host: "127.12.8.61" port: 35001 } attrs { promote: false } } peers { permanent_uuid: "01c12756ade6445f92d995ab935f60d0" member_type: NON_VOTER last_known_addr { host: "127.12.8.60" port: 41857 } attrs { promote: true } }
I20260925 15:13:35.123654 19286 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01c12756ade6445f92d995ab935f60d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c1233ca1ad3745eca6025d08fc59d7f3" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 44383 } } peers { permanent_uuid: "ea8b382304f143f4b4a3dca87c09b7e4" member_type: VOTER last_known_addr { host: "127.12.8.61" port: 35001 } attrs { promote: false } } peers { permanent_uuid: "01c12756ade6445f92d995ab935f60d0" member_type: NON_VOTER last_known_addr { host: "127.12.8.60" port: 41857 } attrs { promote: true } } }
I20260925 15:13:35.123992 19283 sys_catalog.cc:564] T 00000000000000000000000000000000 P 01c12756ade6445f92d995ab935f60d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:13:35.124183 19286 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01c12756ade6445f92d995ab935f60d0 [sys.catalog]: This master's current role is: LEARNER
I20260925 15:13:35.137341 19297 catalog_manager.cc:1259] Loaded cluster ID: 5911822a4ce34211bb32228d45a52d8f
I20260925 15:13:35.137455 19297 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 01c12756ade6445f92d995ab935f60d0: loading cluster ID for follower catalog manager: success
I20260925 15:13:35.140975 19297 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 01c12756ade6445f92d995ab935f60d0: acquiring CA information for follower catalog manager: success
I20260925 15:13:35.143237 19297 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 01c12756ade6445f92d995ab935f60d0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 15:13:35.311789 19130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:35.304217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.60:41857 (local address 127.0.0.1:51138)
0925 15:13:35.304396 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:35.304420 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:35.304546 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:35.305333 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:35.305339 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:35.305371 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:13:35.305568 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:35.305581 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:35.307482 (+ 1901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:35.307488 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:35.309365 (+ 1877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:35.309380 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:35.311005 (+ 1625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:35.311013 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:35.311207 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:35.311247 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 15:13:35.311348 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 15:13:35.311495 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260925 15:13:35.311839 19298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:35.304391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51138 (local address 127.12.8.60:41857)
0925 15:13:35.304974 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:35.304979 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:35.305000 (+ 21us) server_negotiation.cc:407] Connection header received
0925 15:13:35.305074 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:35.305079 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:35.305155 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:35.305269 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:13:35.305763 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:35.307328 (+ 1565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:35.309537 (+ 2209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:35.310826 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:35.310956 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:35.311186 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:35.311385 (+ 199us) server_negotiation.cc:299] Negotiation successful
0925 15:13:35.311534 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":318,"threads_started":1}
I20260925 15:13:35.312873 19258 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 01c12756ade6445f92d995ab935f60d0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260925 15:13:35.313462 19286 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01c12756ade6445f92d995ab935f60d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1233ca1ad3745eca6025d08fc59d7f3. Latest consensus state: current_term: 1 leader_uuid: "c1233ca1ad3745eca6025d08fc59d7f3" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c1233ca1ad3745eca6025d08fc59d7f3" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 44383 } } peers { permanent_uuid: "ea8b382304f143f4b4a3dca87c09b7e4" member_type: VOTER last_known_addr { host: "127.12.8.61" port: 35001 } attrs { promote: false } } peers { permanent_uuid: "01c12756ade6445f92d995ab935f60d0" member_type: NON_VOTER last_known_addr { host: "127.12.8.60" port: 41857 } attrs { promote: true } } }
I20260925 15:13:35.313745 19286 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01c12756ade6445f92d995ab935f60d0 [sys.catalog]: This master's current role is: LEARNER
W20260925 15:13:35.327287 19220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:13:35.327595 19220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:13:35.327715 19220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:13:35.335146 19220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:13:35.335243 19220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:13:35.335357 19220 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.1
I20260925 15:13:35.345289 19220 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:36421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/krb5kdc/oryx_127.12.8.1.keytab
--principal=oryx/127.12.8.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.1:46055
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.8.1
--webserver_port=37297
--tserver_master_addrs=127.12.8.62:44383,127.12.8.61:35001,127.12.8.60:41857
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.1
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:13:35.346567 19220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:13:35.347788 19220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:13:35.360458 19303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:35.360435 19304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:35.360385 19306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:13:35.361614 19220 server_base.cc:1034] running on GCE node
I20260925 15:13:35.362304 19220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:13:35.363189 19220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:13:35.364413 19220 hybrid_clock.cc:648] HybridClock initialized: now 1790349215364368 us; error 58 us; skew 500 ppm
I20260925 15:13:35.364717 19220 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:13:35 dist-test-slave-p468 krb5kdc[18415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349215, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.8.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:13:35.367866 19220 init.cc:377] Logged in from keytab as oryx/127.12.8.1@KRBTEST.COM (short username oryx)
I20260925 15:13:35.369309 19220 webserver.cc:460] Webserver started at http://127.12.8.1:37297/ using document root <none> and password file <none>
I20260925 15:13:35.369979 19220 fs_manager.cc:359] Metadata directory not provided
I20260925 15:13:35.370180 19220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:13:35.375444 19220 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 15:13:35.377993 19313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:35.379338 19220 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:13:35.379503 19220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/ts-0/wal
uuid: "372093292e564acaae5d8d6370b2a26c"
format_stamp: "Formatted at 2026-09-25 15:13:32 on dist-test-slave-p468"
server_key: "6a585b49af4fefef266a92b410e4b3c7"
server_key_iv: "00fa0f60d15900959405f90eacff9b10"
server_key_version: "encryptionkey@0"
I20260925 15:13:35.380540 19220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:13:35.396576 19220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:13:35.401015 19220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:13:35.401238 19220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:13:35.402141 19220 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:13:35.403451 19220 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:13:35.403537 19220 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:35.403612 19220 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:13:35.403671 19220 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:35.436259 19220 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.1:46055
I20260925 15:13:35.436339 19425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.1:46055 every 8 connection(s)
I20260925 15:13:35.437683 19220 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/ts-0/data/info.pb
I20260925 15:13:35.441704 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 19220
I20260925 15:13:35.442049 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 18656
Sep 25 15:13:35 dist-test-slave-p468 krb5kdc[18415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349215, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.8.1@KRBTEST.COM for oryx/127.12.8.60@KRBTEST.COM
Sep 25 15:13:35 dist-test-slave-p468 krb5kdc[18415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349215, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.8.1@KRBTEST.COM for oryx/127.12.8.61@KRBTEST.COM
Sep 25 15:13:35 dist-test-slave-p468 krb5kdc[18415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349215, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.8.1@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM
I20260925 15:13:35.476961 19298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:35.440594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.1:43943 (local address 127.12.8.60:41857)
0925 15:13:35.440817 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:35.440824 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:35.452758 (+ 11934us) server_negotiation.cc:407] Connection header received
0925 15:13:35.454169 (+ 1411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:35.454178 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:35.454308 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:35.454420 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:35.457895 (+ 3475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:35.459262 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:35.462330 (+ 3068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:35.463011 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:35.463108 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:35.468277 (+ 5169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:35.468285 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:35.468291 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:35.468340 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:35.472865 (+ 4525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:35.473889 (+ 1024us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:35.473895 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:35.473901 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:35.474047 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:35.474655 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:35.474661 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:35.474666 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:35.475067 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:35.475184 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:35.476292 (+ 1108us) server_negotiation.cc:299] Negotiation successful
0925 15:13:35.476559 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 15:13:35.478935 19431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:35.441820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.60:41857 (local address 127.12.8.1:43943)
0925 15:13:35.452638 (+ 10818us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:35.452659 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:35.453966 (+ 1307us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:35.455186 (+ 1220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:35.455204 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:35.456465 (+ 1261us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:13:35.457629 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:35.457660 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:35.459519 (+ 1859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:35.459528 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:35.462125 (+ 2597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:35.462140 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:35.463484 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:35.463492 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:35.463773 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:35.464683 (+ 910us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:13:35.464720 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:13:35.468023 (+ 3303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:13:35.473016 (+ 4993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:13:35.473025 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:13:35.473043 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:13:35.473716 (+ 673us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:13:35.474220 (+ 504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:13:35.474228 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:13:35.474234 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:13:35.474508 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:13:35.475300 (+ 792us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:13:35.475310 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:13:35.475823 (+ 513us) client_negotiation.cc:769] Sending connection context
0925 15:13:35.477252 (+ 1429us) client_negotiation.cc:240] Negotiation successful
0925 15:13:35.477564 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10676,"thread_start_us":10503,"threads_started":1}
I20260925 15:13:35.478979 19435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:35.458926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.61:35001 (local address 127.12.8.1:33437)
0925 15:13:35.460758 (+ 1832us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:35.460777 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:35.460890 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:35.461618 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:35.461623 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:35.462295 (+ 672us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:13:35.462684 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:35.462694 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:35.464092 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:35.464099 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:35.465678 (+ 1579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:35.465694 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:35.467658 (+ 1964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:35.467665 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:35.467862 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:35.468528 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:13:35.468544 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:13:35.470728 (+ 2184us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:13:35.474629 (+ 3901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:13:35.474637 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:13:35.474641 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:13:35.475153 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:13:35.475596 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:13:35.475606 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:13:35.475612 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:13:35.475790 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:13:35.476502 (+ 712us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:13:35.476507 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:13:35.476647 (+ 140us) client_negotiation.cc:769] Sending connection context
0925 15:13:35.476815 (+ 168us) client_negotiation.cc:240] Negotiation successful
0925 15:13:35.477387 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1714,"thread_start_us":2475,"threads_started":1}
I20260925 15:13:35.479012 19434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:35.452933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.1:33437 (local address 127.12.8.61:35001)
0925 15:13:35.459276 (+ 6343us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:35.459285 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:35.460993 (+ 1708us) server_negotiation.cc:407] Connection header received
0925 15:13:35.461079 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:35.461085 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:35.461172 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:35.461271 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:35.462855 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:35.463880 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:35.465854 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:35.466576 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:35.466712 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:35.470921 (+ 4209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:35.470928 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:35.470932 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:35.470975 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:35.474467 (+ 3492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:35.475299 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:35.475304 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:35.475307 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:35.475456 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:35.475928 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:35.475935 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:35.475938 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:35.476279 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:35.476436 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:35.478378 (+ 1942us) server_negotiation.cc:299] Negotiation successful
0925 15:13:35.478593 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6178,"thread_start_us":6117,"threads_started":1}
I20260925 15:13:35.480523 19428 heartbeater.cc:347] Connected to a master server at 127.12.8.61:35001
I20260925 15:13:35.480672 19426 heartbeater.cc:347] Connected to a master server at 127.12.8.60:41857
I20260925 15:13:35.480825 19428 heartbeater.cc:464] Registering TS with master...
I20260925 15:13:35.480830 19426 heartbeater.cc:464] Registering TS with master...
I20260925 15:13:35.481814 19428 heartbeater.cc:511] Master 127.12.8.61:35001 requested a full tablet report, sending...
Sep 25 15:13:35 dist-test-slave-p468 krb5kdc[18415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349211, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.8.2@KRBTEST.COM
I20260925 15:13:35.484243 19430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:35.441192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:44383 (local address 127.12.8.1:39779)
0925 15:13:35.442411 (+ 1219us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:35.442477 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:35.452334 (+ 9857us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:35.457164 (+ 4830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:35.457171 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:35.457806 (+ 635us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:13:35.458162 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:35.458176 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:35.461794 (+ 3618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:35.461802 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:35.464041 (+ 2239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:35.464060 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:35.473165 (+ 9105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:35.473173 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:35.473390 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:35.474324 (+ 934us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:13:35.474348 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:13:35.477101 (+ 2753us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:13:35.481146 (+ 4045us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:13:35.481151 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:13:35.481154 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:13:35.481684 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:13:35.482164 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:13:35.482168 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:13:35.482171 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:13:35.482364 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:13:35.483153 (+ 789us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:13:35.483158 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:13:35.483313 (+ 155us) client_negotiation.cc:769] Sending connection context
0925 15:13:35.483461 (+ 148us) client_negotiation.cc:240] Negotiation successful
0925 15:13:35.483856 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":728,"spinlock_wait_cycles":1408,"thread_start_us":730,"threads_started":1}
I20260925 15:13:35.484359 19058 ts_manager.cc:194] Registered new tserver with Master: 372093292e564acaae5d8d6370b2a26c (127.12.8.1:46055)
I20260925 15:13:35.484760 18652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:35.440464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.1:39779 (local address 127.12.8.62:44383)
0925 15:13:35.440683 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:35.440690 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:35.442619 (+ 1929us) server_negotiation.cc:407] Connection header received
0925 15:13:35.456635 (+ 14016us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:35.456644 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:35.456761 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:35.456853 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:35.460053 (+ 3200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:35.461504 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:35.470366 (+ 8862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:35.471034 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:35.471131 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:35.477342 (+ 6211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:35.477349 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:35.477353 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:35.477397 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:35.480996 (+ 3599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:35.481850 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:35.481855 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:35.481858 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:35.482012 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:35.482543 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:35.482549 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:35.482552 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:35.482921 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:35.483062 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:35.483616 (+ 554us) server_negotiation.cc:299] Negotiation successful
0925 15:13:35.483940 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260925 15:13:35.484958 19426 heartbeater.cc:511] Master 127.12.8.60:41857 requested a full tablet report, sending...
I20260925 15:13:35.485811 19427 heartbeater.cc:347] Connected to a master server at 127.12.8.62:44383
I20260925 15:13:35.485939 19427 heartbeater.cc:464] Registering TS with master...
I20260925 15:13:35.486435 19427 heartbeater.cc:511] Master 127.12.8.62:44383 requested a full tablet report, sending...
I20260925 15:13:35.487226 19248 ts_manager.cc:194] Registered new tserver with Master: 372093292e564acaae5d8d6370b2a26c (127.12.8.1:46055)
I20260925 15:13:35.487577 18460 ts_manager.cc:194] Re-registered known tserver with Master: 372093292e564acaae5d8d6370b2a26c (127.12.8.1:46055)
I20260925 15:13:35.488626 18460 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.8.1@KRBTEST.COM'} at 127.12.8.1:39779
I20260925 15:13:35.491413 19433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:35.443147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41370 (local address 127.12.8.2:35849)
0925 15:13:35.466091 (+ 22944us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:35.466104 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:35.466141 (+ 37us) server_negotiation.cc:407] Connection header received
0925 15:13:35.466508 (+ 367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:35.466521 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:35.466869 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:35.469684 (+ 2815us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:35.469949 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:35.471491 (+ 1542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:35.472960 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:35.474070 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:35.482504 (+ 8434us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:35.484198 (+ 1694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:35.484215 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:35.484232 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:35.484280 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:35.488375 (+ 4095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:35.489101 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:35.489110 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:35.489113 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:35.489242 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:35.489760 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:35.489765 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:35.489768 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:35.490170 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:35.490536 (+ 366us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:35.490804 (+ 268us) server_negotiation.cc:299] Negotiation successful
0925 15:13:35.491059 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22696,"thread_start_us":23045,"threads_started":1}
I20260925 15:13:35.492517 18779 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:13:35.576124 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 18656
I20260925 15:13:35.592412 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.2:35849
--local_ip_for_outbound_sockets=127.12.8.2
--tserver_master_addrs=127.12.8.62:44383
--webserver_port=41793
--webserver_interface=127.12.8.2
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/krb5kdc/oryx_127.12.8.2.keytab
--principal=oryx/127.12.8.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:36421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.12.8.62:44383,127.12.8.61:35001
--tserver_master_addrs=127.12.8.62:44383,127.12.8.61:35001,127.12.8.60:41857 with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:13:35.779218 19212 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1233ca1ad3745eca6025d08fc59d7f3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c1233ca1ad3745eca6025d08fc59d7f3" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c1233ca1ad3745eca6025d08fc59d7f3" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 44383 } } peers { permanent_uuid: "ea8b382304f143f4b4a3dca87c09b7e4" member_type: VOTER last_known_addr { host: "127.12.8.61" port: 35001 } attrs { promote: false } } peers { permanent_uuid: "01c12756ade6445f92d995ab935f60d0" member_type: NON_VOTER last_known_addr { host: "127.12.8.60" port: 41857 } attrs { promote: true } } }
I20260925 15:13:35.779644 19212 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1233ca1ad3745eca6025d08fc59d7f3 [sys.catalog]: This master's current role is: LEADER
I20260925 15:13:35.781064 19212 raft_consensus.cc:1050] T 00000000000000000000000000000000 P c1233ca1ad3745eca6025d08fc59d7f3: attempting to promote NON_VOTER 01c12756ade6445f92d995ab935f60d0 to VOTER
I20260925 15:13:35.781728 19212 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1233ca1ad3745eca6025d08fc59d7f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c1233ca1ad3745eca6025d08fc59d7f3" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 44383 } } peers { permanent_uuid: "ea8b382304f143f4b4a3dca87c09b7e4" member_type: VOTER last_known_addr { host: "127.12.8.61" port: 35001 } attrs { promote: false } } peers { permanent_uuid: "01c12756ade6445f92d995ab935f60d0" member_type: VOTER last_known_addr { host: "127.12.8.60" port: 41857 } attrs { promote: false } }
I20260925 15:13:35.783788 19258 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 01c12756ade6445f92d995ab935f60d0 [term 1 LEARNER]: Refusing update from remote peer c1233ca1ad3745eca6025d08fc59d7f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260925 15:13:35.783929 19068 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ea8b382304f143f4b4a3dca87c09b7e4 [term 1 FOLLOWER]: Refusing update from remote peer c1233ca1ad3745eca6025d08fc59d7f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260925 15:13:35.784456 19212 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c1233ca1ad3745eca6025d08fc59d7f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01c12756ade6445f92d995ab935f60d0" member_type: VOTER last_known_addr { host: "127.12.8.60" port: 41857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260925 15:13:35.784814 19439 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c1233ca1ad3745eca6025d08fc59d7f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea8b382304f143f4b4a3dca87c09b7e4" member_type: VOTER last_known_addr { host: "127.12.8.61" port: 35001 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260925 15:13:35.787752 19212 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c1233ca1ad3745eca6025d08fc59d7f3 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 01c12756ade6445f92d995ab935f60d0 (127.12.8.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c1233ca1ad3745eca6025d08fc59d7f3" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 44383 } } peers { permanent_uuid: "ea8b382304f143f4b4a3dca87c09b7e4" member_type: VOTER last_known_addr { host: "127.12.8.61" port: 35001 } attrs { promote: false } } peers { permanent_uuid: "01c12756ade6445f92d995ab935f60d0" member_type: VOTER last_known_addr { host: "127.12.8.60" port: 41857 } attrs { promote: false } } }
I20260925 15:13:35.790731 19212 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1233ca1ad3745eca6025d08fc59d7f3 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c1233ca1ad3745eca6025d08fc59d7f3" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c1233ca1ad3745eca6025d08fc59d7f3" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 44383 } } peers { permanent_uuid: "ea8b382304f143f4b4a3dca87c09b7e4" member_type: VOTER last_known_addr { host: "127.12.8.61" port: 35001 } attrs { promote: false } } peers { permanent_uuid: "01c12756ade6445f92d995ab935f60d0" member_type: VOTER last_known_addr { host: "127.12.8.60" port: 41857 } attrs { promote: false } } }
I20260925 15:13:35.790970 19212 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1233ca1ad3745eca6025d08fc59d7f3 [sys.catalog]: This master's current role is: LEADER
I20260925 15:13:35.791218 19438 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1233ca1ad3745eca6025d08fc59d7f3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c1233ca1ad3745eca6025d08fc59d7f3" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c1233ca1ad3745eca6025d08fc59d7f3" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 44383 } } peers { permanent_uuid: "ea8b382304f143f4b4a3dca87c09b7e4" member_type: VOTER last_known_addr { host: "127.12.8.61" port: 35001 } attrs { promote: false } } peers { permanent_uuid: "01c12756ade6445f92d995ab935f60d0" member_type: VOTER last_known_addr { host: "127.12.8.60" port: 41857 } attrs { promote: false } } }
I20260925 15:13:35.791460 19438 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1233ca1ad3745eca6025d08fc59d7f3 [sys.catalog]: This master's current role is: LEADER
I20260925 15:13:35.790930 19068 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ea8b382304f143f4b4a3dca87c09b7e4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 01c12756ade6445f92d995ab935f60d0 (127.12.8.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c1233ca1ad3745eca6025d08fc59d7f3" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 44383 } } peers { permanent_uuid: "ea8b382304f143f4b4a3dca87c09b7e4" member_type: VOTER last_known_addr { host: "127.12.8.61" port: 35001 } attrs { promote: false } } peers { permanent_uuid: "01c12756ade6445f92d995ab935f60d0" member_type: VOTER last_known_addr { host: "127.12.8.60" port: 41857 } attrs { promote: false } } }
I20260925 15:13:35.792052 19212 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1233ca1ad3745eca6025d08fc59d7f3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c1233ca1ad3745eca6025d08fc59d7f3" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c1233ca1ad3745eca6025d08fc59d7f3" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 44383 } } peers { permanent_uuid: "ea8b382304f143f4b4a3dca87c09b7e4" member_type: VOTER last_known_addr { host: "127.12.8.61" port: 35001 } attrs { promote: false } } peers { permanent_uuid: "01c12756ade6445f92d995ab935f60d0" member_type: VOTER last_known_addr { host: "127.12.8.60" port: 41857 } attrs { promote: false } } }
I20260925 15:13:35.792260 19212 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1233ca1ad3745eca6025d08fc59d7f3 [sys.catalog]: This master's current role is: LEADER
I20260925 15:13:35.793298 19258 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 01c12756ade6445f92d995ab935f60d0 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 01c12756ade6445f92d995ab935f60d0 (127.12.8.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c1233ca1ad3745eca6025d08fc59d7f3" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 44383 } } peers { permanent_uuid: "ea8b382304f143f4b4a3dca87c09b7e4" member_type: VOTER last_known_addr { host: "127.12.8.61" port: 35001 } attrs { promote: false } } peers { permanent_uuid: "01c12756ade6445f92d995ab935f60d0" member_type: VOTER last_known_addr { host: "127.12.8.60" port: 41857 } attrs { promote: false } } }
I20260925 15:13:35.798447 19286 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01c12756ade6445f92d995ab935f60d0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c1233ca1ad3745eca6025d08fc59d7f3" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c1233ca1ad3745eca6025d08fc59d7f3" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 44383 } } peers { permanent_uuid: "ea8b382304f143f4b4a3dca87c09b7e4" member_type: VOTER last_known_addr { host: "127.12.8.61" port: 35001 } attrs { promote: false } } peers { permanent_uuid: "01c12756ade6445f92d995ab935f60d0" member_type: VOTER last_known_addr { host: "127.12.8.60" port: 41857 } attrs { promote: false } } }
I20260925 15:13:35.798691 19286 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01c12756ade6445f92d995ab935f60d0 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:13:35.797996 19442 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea8b382304f143f4b4a3dca87c09b7e4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c1233ca1ad3745eca6025d08fc59d7f3" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c1233ca1ad3745eca6025d08fc59d7f3" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 44383 } } peers { permanent_uuid: "ea8b382304f143f4b4a3dca87c09b7e4" member_type: VOTER last_known_addr { host: "127.12.8.61" port: 35001 } attrs { promote: false } } peers { permanent_uuid: "01c12756ade6445f92d995ab935f60d0" member_type: VOTER last_known_addr { host: "127.12.8.60" port: 41857 } attrs { promote: false } } }
I20260925 15:13:35.799011 19442 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea8b382304f143f4b4a3dca87c09b7e4 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 15:13:35.918565 19437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:13:35.918871 19437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:13:35.919046 19437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:13:35.926430 19437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:13:35.926568 19437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:13:35.926730 19437 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.2
I20260925 15:13:35.936456 19437 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:36421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/krb5kdc/oryx_127.12.8.2.keytab
--principal=oryx/127.12.8.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.2:35849
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.8.2
--webserver_port=41793
--tserver_master_addrs=127.12.8.62:44383,127.12.8.61:35001,127.12.8.60:41857
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.2
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:13:35.937722 19437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:13:35.939034 19437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:13:35.952848 19447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:35.952869 19448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:35.952797 19450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:13:35.953655 19437 server_base.cc:1034] running on GCE node
I20260925 15:13:35.954293 19437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:13:35.955181 19437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:13:35.956408 19437 hybrid_clock.cc:648] HybridClock initialized: now 1790349215956368 us; error 54 us; skew 500 ppm
I20260925 15:13:35.956758 19437 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:13:35 dist-test-slave-p468 krb5kdc[18415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349215, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.8.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:13:35.959811 19437 init.cc:377] Logged in from keytab as oryx/127.12.8.2@KRBTEST.COM (short username oryx)
I20260925 15:13:35.961191 19437 webserver.cc:460] Webserver started at http://127.12.8.2:41793/ using document root <none> and password file <none>
I20260925 15:13:35.961854 19437 fs_manager.cc:359] Metadata directory not provided
I20260925 15:13:35.962009 19437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:13:35.969122 19437 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.000s sys 0.004s
I20260925 15:13:35.971832 19457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:35.973372 19437 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 15:13:35.973533 19437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/ts-1/wal
uuid: "ce72ef070a95404f8dbb29d80d7cff69"
format_stamp: "Formatted at 2026-09-25 15:13:32 on dist-test-slave-p468"
server_key: "5266384aaf8008ac737a5dfe4010ef1f"
server_key_iv: "96c15a2a7e04a251d48a29691874d141"
server_key_version: "encryptionkey@0"
I20260925 15:13:35.974551 19437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:13:35.995653 19437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:13:36.000537 19437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:13:36.000829 19437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:13:36.001778 19437 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:13:36.003144 19437 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:13:36.003252 19437 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:36.003381 19437 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:13:36.003484 19437 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:36.037792 19437 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.2:35849
I20260925 15:13:36.037925 19569 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.2:35849 every 8 connection(s)
I20260925 15:13:36.039336 19437 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/ts-1/data/info.pb
I20260925 15:13:36.041417 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 19437
I20260925 15:13:36.041743 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 18789
Sep 25 15:13:36 dist-test-slave-p468 krb5kdc[18415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349215, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.8.2@KRBTEST.COM for oryx/127.12.8.61@KRBTEST.COM
Sep 25 15:13:36 dist-test-slave-p468 krb5kdc[18415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349215, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.8.2@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM
Sep 25 15:13:36 dist-test-slave-p468 krb5kdc[18415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349215, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.8.2@KRBTEST.COM for oryx/127.12.8.60@KRBTEST.COM
I20260925 15:13:36.070256 19574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:36.042767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.2:55367 (local address 127.12.8.61:35001)
0925 15:13:36.044110 (+ 1343us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:36.044118 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:36.044990 (+ 872us) server_negotiation.cc:407] Connection header received
0925 15:13:36.048192 (+ 3202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:36.048198 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:36.048293 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:36.048390 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:36.051101 (+ 2711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:36.052377 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:36.055479 (+ 3102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:36.056100 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:36.056194 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:36.061156 (+ 4962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:36.061163 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:36.061168 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:36.061213 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:36.066433 (+ 5220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:36.067352 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:36.067357 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:36.067361 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:36.067534 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:36.068090 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:36.068096 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:36.068101 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:36.068484 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:36.068653 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:36.069552 (+ 899us) server_negotiation.cc:299] Negotiation successful
0925 15:13:36.069840 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1170,"thread_start_us":998,"threads_started":1}
I20260925 15:13:36.072796 19581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:36.052796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.2:44277 (local address 127.12.8.60:41857)
0925 15:13:36.055465 (+ 2669us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:36.055471 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:36.056079 (+ 608us) server_negotiation.cc:407] Connection header received
0925 15:13:36.056151 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:36.056159 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:36.056237 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:36.056342 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:36.057736 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:36.058904 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:36.060820 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:36.061509 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:36.061620 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:36.065533 (+ 3913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:36.065539 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:36.065544 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:36.065586 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:36.069008 (+ 3422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:36.070236 (+ 1228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:36.070243 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:36.070248 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:36.070670 (+ 422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:36.071142 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:36.071147 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:36.071151 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:36.071456 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:36.071565 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:36.072175 (+ 610us) server_negotiation.cc:299] Negotiation successful
0925 15:13:36.072385 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2559,"thread_start_us":2459,"threads_started":1}
I20260925 15:13:36.073673 19578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:36.044778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.2:48079 (local address 127.12.8.62:44383)
0925 15:13:36.052311 (+ 7533us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:36.052320 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:36.052344 (+ 24us) server_negotiation.cc:407] Connection header received
0925 15:13:36.052418 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:36.052427 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:36.052529 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:36.052658 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:36.054427 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:36.055710 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:36.058323 (+ 2613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:36.059282 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:36.059434 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:36.066266 (+ 6832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:36.066273 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:36.066278 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:36.066588 (+ 310us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:36.069868 (+ 3280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:36.070699 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:36.070704 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:36.070708 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:36.070843 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:36.071621 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:36.071625 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:36.071627 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:36.071852 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:36.072007 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:36.073092 (+ 1085us) server_negotiation.cc:299] Negotiation successful
0925 15:13:36.073315 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7357,"thread_start_us":7231,"threads_started":1}
I20260925 15:13:36.073931 19580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:36.052432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.60:41857 (local address 127.12.8.2:44277)
0925 15:13:36.055898 (+ 3466us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:36.055915 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:36.056048 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:36.056528 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:36.056533 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:36.057137 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:13:36.057546 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:36.057561 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:36.059126 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:36.059132 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:36.060638 (+ 1506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:36.060651 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:36.061936 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:36.061942 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:36.062164 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:36.062828 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:13:36.062853 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:13:36.065325 (+ 2472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:13:36.069159 (+ 3834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:13:36.069164 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:13:36.069167 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:13:36.069674 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:13:36.070828 (+ 1154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:13:36.070833 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:13:36.070836 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:13:36.071016 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:13:36.071598 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:13:36.071605 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:13:36.071912 (+ 307us) client_negotiation.cc:769] Sending connection context
0925 15:13:36.072095 (+ 183us) client_negotiation.cc:240] Negotiation successful
0925 15:13:36.072381 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3343,"thread_start_us":3175,"threads_started":1}
Sep 25 15:13:36 dist-test-slave-p468 krb5kdc[18415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349211, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.8.3@KRBTEST.COM
I20260925 15:13:36.074592 19579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:36.043582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:44383 (local address 127.12.8.2:48079)
0925 15:13:36.051612 (+ 8030us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:36.051636 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:36.051775 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:36.052879 (+ 1104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:36.052887 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:36.053727 (+ 840us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:13:36.054159 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:36.054175 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:36.055886 (+ 1711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:36.055891 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:36.058076 (+ 2185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:36.058096 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:36.060843 (+ 2747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:36.060852 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:36.061021 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:36.061655 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:13:36.061681 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:13:36.066018 (+ 4337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:13:36.070015 (+ 3997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:13:36.070020 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:13:36.070023 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:13:36.070573 (+ 550us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:13:36.070962 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:13:36.070968 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:13:36.070971 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:13:36.071210 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:13:36.071973 (+ 763us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:13:36.071978 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:13:36.072135 (+ 157us) client_negotiation.cc:769] Sending connection context
0925 15:13:36.072259 (+ 124us) client_negotiation.cc:240] Negotiation successful
0925 15:13:36.072521 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7838,"mutex_wait_us":3223,"spinlock_wait_cycles":20736,"thread_start_us":4393,"threads_started":1}
I20260925 15:13:36.075662 19571 heartbeater.cc:347] Connected to a master server at 127.12.8.62:44383
I20260925 15:13:36.075698 19570 heartbeater.cc:347] Connected to a master server at 127.12.8.60:41857
I20260925 15:13:36.076009 19571 heartbeater.cc:464] Registering TS with master...
I20260925 15:13:36.076028 19570 heartbeater.cc:464] Registering TS with master...
I20260925 15:13:36.076871 19575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:36.043582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.61:35001 (local address 127.12.8.2:55367)
0925 15:13:36.044660 (+ 1078us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:36.044714 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:36.047878 (+ 3164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:36.049031 (+ 1153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:36.049044 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:36.049904 (+ 860us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:13:36.050846 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:36.050886 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:36.052618 (+ 1732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:36.052623 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:36.055285 (+ 2662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:36.055301 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:36.056563 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:36.056572 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:36.056915 (+ 343us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:36.057766 (+ 851us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:13:36.057800 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:13:36.060936 (+ 3136us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:13:36.066614 (+ 5678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:13:36.066623 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:13:36.066640 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:13:36.067207 (+ 567us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:13:36.067689 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:13:36.067694 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:13:36.067697 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:13:36.067939 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:13:36.068637 (+ 698us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:13:36.068646 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:13:36.069085 (+ 439us) client_negotiation.cc:769] Sending connection context
0925 15:13:36.069412 (+ 327us) client_negotiation.cc:240] Negotiation successful
0925 15:13:36.069972 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":756,"thread_start_us":3731,"threads_started":1}
I20260925 15:13:36.077099 19570 heartbeater.cc:511] Master 127.12.8.60:41857 requested a full tablet report, sending...
I20260925 15:13:36.077103 19571 heartbeater.cc:511] Master 127.12.8.62:44383 requested a full tablet report, sending...
I20260925 15:13:36.078334 19572 heartbeater.cc:347] Connected to a master server at 127.12.8.61:35001
I20260925 15:13:36.078451 19572 heartbeater.cc:464] Registering TS with master...
I20260925 15:13:36.078831 19572 heartbeater.cc:511] Master 127.12.8.61:35001 requested a full tablet report, sending...
I20260925 15:13:36.078869 18460 ts_manager.cc:194] Re-registered known tserver with Master: ce72ef070a95404f8dbb29d80d7cff69 (127.12.8.2:35849)
I20260925 15:13:36.079088 19248 ts_manager.cc:194] Registered new tserver with Master: ce72ef070a95404f8dbb29d80d7cff69 (127.12.8.2:35849)
I20260925 15:13:36.079875 19058 ts_manager.cc:194] Registered new tserver with Master: ce72ef070a95404f8dbb29d80d7cff69 (127.12.8.2:35849)
I20260925 15:13:36.080029 18460 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.8.2@KRBTEST.COM'} at 127.12.8.2:48079
I20260925 15:13:36.083498 19577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:36.044648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51898 (local address 127.12.8.3:40103)
0925 15:13:36.060090 (+ 15442us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:36.060098 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:36.060123 (+ 25us) server_negotiation.cc:407] Connection header received
0925 15:13:36.060176 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:36.060182 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:36.060365 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:36.060535 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:36.062714 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:36.064014 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:36.065389 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:36.066354 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:36.066490 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:36.075861 (+ 9371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:36.075875 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:36.075889 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:36.075938 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:36.080648 (+ 4710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:36.081458 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:36.081464 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:36.081466 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:36.081611 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:36.082016 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:36.082019 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:36.082021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:36.082390 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:36.082612 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:36.082934 (+ 322us) server_negotiation.cc:299] Negotiation successful
0925 15:13:36.083120 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15266,"thread_start_us":15109,"threads_started":1}
I20260925 15:13:36.084726 18912 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:13:36.168754 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 18789
I20260925 15:13:36.185961 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.3:40103
--local_ip_for_outbound_sockets=127.12.8.3
--tserver_master_addrs=127.12.8.62:44383
--webserver_port=44911
--webserver_interface=127.12.8.3
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/krb5kdc/oryx_127.12.8.3.keytab
--principal=oryx/127.12.8.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:36421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.12.8.62:44383,127.12.8.61:35001
--tserver_master_addrs=127.12.8.62:44383,127.12.8.61:35001,127.12.8.60:41857 with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:13:36.491276 19427 heartbeater.cc:503] Master 127.12.8.62:44383 was elected leader, sending a full tablet report...
W20260925 15:13:36.514340 19583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:13:36.514603 19583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:13:36.514765 19583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:13:36.522025 19583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:13:36.522150 19583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:13:36.522358 19583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.3
I20260925 15:13:36.531906 19583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:36421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/krb5kdc/oryx_127.12.8.3.keytab
--principal=oryx/127.12.8.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.3:40103
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.8.3
--webserver_port=44911
--tserver_master_addrs=127.12.8.62:44383,127.12.8.61:35001,127.12.8.60:41857
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.3
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:13:36.533135 19583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:13:36.534475 19583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:13:36.547303 19591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:36.547336 19589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:36.547695 19588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:13:36.548156 19583 server_base.cc:1034] running on GCE node
I20260925 15:13:36.548715 19583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:13:36.549571 19583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:13:36.550788 19583 hybrid_clock.cc:648] HybridClock initialized: now 1790349216550751 us; error 48 us; skew 500 ppm
I20260925 15:13:36.551097 19583 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:13:36 dist-test-slave-p468 krb5kdc[18415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349216, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.8.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:13:36.554106 19583 init.cc:377] Logged in from keytab as oryx/127.12.8.3@KRBTEST.COM (short username oryx)
I20260925 15:13:36.555500 19583 webserver.cc:460] Webserver started at http://127.12.8.3:44911/ using document root <none> and password file <none>
I20260925 15:13:36.556159 19583 fs_manager.cc:359] Metadata directory not provided
I20260925 15:13:36.556316 19583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:13:36.561376 19583 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 15:13:36.563905 19598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:36.565095 19583 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 15:13:36.565266 19583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/ts-2/wal
uuid: "998c5926ac2741278b4a9c80cd3f8df5"
format_stamp: "Formatted at 2026-09-25 15:13:33 on dist-test-slave-p468"
server_key: "b0c0c690b2a3a4dc61f5e9277b13b03c"
server_key_iv: "eda382e2585032d5f393a671dabac984"
server_key_version: "encryptionkey@0"
I20260925 15:13:36.566190 19583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:13:36.582103 19583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:13:36.586364 19583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:13:36.586644 19583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:13:36.587523 19583 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:13:36.588770 19583 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:13:36.588882 19583 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:36.589001 19583 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:13:36.589098 19583 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:36.619827 19583 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.3:40103
I20260925 15:13:36.619925 19710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.3:40103 every 8 connection(s)
I20260925 15:13:36.621232 19583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/ts-2/data/info.pb
I20260925 15:13:36.626381 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 19583
Sep 25 15:13:36 dist-test-slave-p468 krb5kdc[18415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349216, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.8.3@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM
Sep 25 15:13:36 dist-test-slave-p468 krb5kdc[18415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349216, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.8.3@KRBTEST.COM for oryx/127.12.8.60@KRBTEST.COM
Sep 25 15:13:36 dist-test-slave-p468 krb5kdc[18415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349216, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.8.3@KRBTEST.COM for oryx/127.12.8.61@KRBTEST.COM
I20260925 15:13:36.656538 19718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:36.625492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:44383 (local address 127.12.8.3:39505)
0925 15:13:36.626114 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:36.626133 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:36.628363 (+ 2230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:36.629333 (+ 970us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:36.629344 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:36.630558 (+ 1214us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:13:36.631513 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:36.631540 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:36.636479 (+ 4939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:36.636486 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:36.638515 (+ 2029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:36.638529 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:36.640437 (+ 1908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:36.640445 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:36.640689 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:36.641475 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:13:36.641507 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:13:36.647144 (+ 5637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:13:36.651454 (+ 4310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:13:36.651467 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:13:36.651490 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:13:36.652127 (+ 637us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:13:36.652625 (+ 498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:13:36.652633 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:13:36.652639 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:13:36.652975 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:13:36.653701 (+ 726us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:13:36.653709 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:13:36.654207 (+ 498us) client_negotiation.cc:769] Sending connection context
0925 15:13:36.654531 (+ 324us) client_negotiation.cc:240] Negotiation successful
0925 15:13:36.655091 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"mutex_wait_us":4,"thread_start_us":2025,"threads_started":1}
I20260925 15:13:36.656919 19719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:36.626906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.60:41857 (local address 127.12.8.3:38277)
0925 15:13:36.627420 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:36.627436 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:36.628363 (+ 927us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:36.629333 (+ 970us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:36.629344 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:36.631742 (+ 2398us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:13:36.632048 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:36.632058 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:36.633578 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:36.633586 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:36.636288 (+ 2702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:36.636307 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:36.639752 (+ 3445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:36.639761 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:36.640203 (+ 442us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:36.643380 (+ 3177us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:13:36.643410 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:13:36.647548 (+ 4138us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:13:36.652422 (+ 4874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:13:36.652430 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:13:36.652437 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:13:36.653122 (+ 685us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:13:36.653634 (+ 512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:13:36.653642 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:13:36.653648 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:13:36.653841 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:13:36.655312 (+ 1471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:13:36.655320 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:13:36.655507 (+ 187us) client_negotiation.cc:769] Sending connection context
0925 15:13:36.655671 (+ 164us) client_negotiation.cc:240] Negotiation successful
0925 15:13:36.656003 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"mutex_wait_us":764,"thread_start_us":795,"threads_started":1}
I20260925 15:13:36.657619 19717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:36.625191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.3:39505 (local address 127.12.8.62:44383)
0925 15:13:36.625607 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:36.625612 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:36.626240 (+ 628us) server_negotiation.cc:407] Connection header received
0925 15:13:36.628757 (+ 2517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:36.628763 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:36.628848 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:36.628947 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:36.631698 (+ 2751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:36.632949 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:36.638649 (+ 5700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:36.639259 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:36.639388 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:36.647368 (+ 7980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:36.647377 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:36.647384 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:36.647425 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:36.651255 (+ 3830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:36.652298 (+ 1043us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:36.652302 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:36.652305 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:36.652470 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:36.653130 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:36.653135 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:36.653138 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:36.653488 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:36.653624 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:36.657033 (+ 3409us) server_negotiation.cc:299] Negotiation successful
0925 15:13:36.657266 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":466,"threads_started":1}
I20260925 15:13:36.658553 19720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:36.627299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.3:38277 (local address 127.12.8.60:41857)
0925 15:13:36.627844 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:36.627850 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:36.627868 (+ 18us) server_negotiation.cc:407] Connection header received
0925 15:13:36.628757 (+ 889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:36.628763 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:36.628844 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:36.628947 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:36.632215 (+ 3268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:36.633368 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:36.638722 (+ 5354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:36.639347 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:36.639424 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:36.647748 (+ 8324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:36.647755 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:36.647761 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:36.647800 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:36.652226 (+ 4426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:36.653295 (+ 1069us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:36.653303 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:36.653309 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:36.653487 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:36.654343 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:36.654351 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:36.654357 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:36.654833 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:36.657459 (+ 2626us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:36.657607 (+ 148us) server_negotiation.cc:299] Negotiation successful
0925 15:13:36.658045 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":485,"threads_started":1}
I20260925 15:13:36.659241 19712 heartbeater.cc:347] Connected to a master server at 127.12.8.62:44383
I20260925 15:13:36.659595 19712 heartbeater.cc:464] Registering TS with master...
I20260925 15:13:36.660558 19712 heartbeater.cc:511] Master 127.12.8.62:44383 requested a full tablet report, sending...
I20260925 15:13:36.661216 19715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:36.624140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.3:36619 (local address 127.12.8.61:35001)
0925 15:13:36.624574 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:36.624578 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:36.625782 (+ 1204us) server_negotiation.cc:407] Connection header received
0925 15:13:36.629264 (+ 3482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:36.629272 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:36.629365 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:36.629461 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:36.637945 (+ 8484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:36.639494 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:36.644654 (+ 5160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:36.645364 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:36.645489 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:36.653190 (+ 7701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:36.653198 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:36.653206 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:36.653254 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:36.656659 (+ 3405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:36.657880 (+ 1221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:36.657886 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:36.657889 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:36.658065 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:36.659623 (+ 1558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:36.659631 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:36.659637 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:36.660095 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:36.660227 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:36.660638 (+ 411us) server_negotiation.cc:299] Negotiation successful
0925 15:13:36.660830 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":233,"threads_started":1}
I20260925 15:13:36.661235 19716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:36.624724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.61:35001 (local address 127.12.8.3:36619)
0925 15:13:36.625608 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:36.625656 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:36.629078 (+ 3422us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:36.629681 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:36.629685 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:36.636616 (+ 6931us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:13:36.637480 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:36.637497 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:36.639761 (+ 2264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:36.639769 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:36.641993 (+ 2224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:36.642010 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:36.647049 (+ 5039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:36.647056 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:36.647270 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:36.648247 (+ 977us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:13:36.648278 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:13:36.652232 (+ 3954us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:13:36.657158 (+ 4926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:13:36.657163 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:13:36.657170 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:13:36.657597 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:13:36.658524 (+ 927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:13:36.658528 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:13:36.658531 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:13:36.659495 (+ 964us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:13:36.660212 (+ 717us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:13:36.660216 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:13:36.660388 (+ 172us) client_negotiation.cc:769] Sending connection context
0925 15:13:36.660540 (+ 152us) client_negotiation.cc:240] Negotiation successful
0925 15:13:36.660813 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"mutex_wait_us":4,"thread_start_us":902,"threads_started":1}
I20260925 15:13:36.661445 19711 heartbeater.cc:347] Connected to a master server at 127.12.8.60:41857
I20260925 15:13:36.661586 19711 heartbeater.cc:464] Registering TS with master...
I20260925 15:13:36.661989 19711 heartbeater.cc:511] Master 127.12.8.60:41857 requested a full tablet report, sending...
I20260925 15:13:36.662079 19713 heartbeater.cc:347] Connected to a master server at 127.12.8.61:35001
I20260925 15:13:36.662215 19713 heartbeater.cc:464] Registering TS with master...
I20260925 15:13:36.662978 19713 heartbeater.cc:511] Master 127.12.8.61:35001 requested a full tablet report, sending...
I20260925 15:13:36.663125 18460 ts_manager.cc:194] Re-registered known tserver with Master: 998c5926ac2741278b4a9c80cd3f8df5 (127.12.8.3:40103)
I20260925 15:13:36.664098 19248 ts_manager.cc:194] Registered new tserver with Master: 998c5926ac2741278b4a9c80cd3f8df5 (127.12.8.3:40103)
I20260925 15:13:36.664275 18460 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.8.3@KRBTEST.COM'} at 127.12.8.3:39505
I20260925 15:13:36.665010 19058 ts_manager.cc:194] Registered new tserver with Master: 998c5926ac2741278b4a9c80cd3f8df5 (127.12.8.3:40103)
I20260925 15:13:36.682096 19717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:36.630555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60696 (local address 127.12.8.62:44383)
0925 15:13:36.658020 (+ 27465us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:36.658025 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:36.658042 (+ 17us) server_negotiation.cc:407] Connection header received
0925 15:13:36.658085 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:36.658092 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:36.658176 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:36.658282 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:36.662683 (+ 4401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:36.664519 (+ 1836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:36.668933 (+ 4414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:36.669820 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:36.669993 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:36.674536 (+ 4543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:36.674544 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:36.674553 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:36.674583 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:36.678249 (+ 3666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:36.679198 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:36.679202 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:36.679204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:36.679356 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:36.680211 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:36.680222 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:36.680225 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:36.680551 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:36.680792 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:36.681472 (+ 680us) server_negotiation.cc:299] Negotiation successful
0925 15:13:36.681693 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27345,"thread_start_us":40878,"threads_started":1}
I20260925 15:13:36.687863 19715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:36.630615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49510 (local address 127.12.8.61:35001)
0925 15:13:36.662037 (+ 31422us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:36.662045 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:36.662160 (+ 115us) server_negotiation.cc:407] Connection header received
0925 15:13:36.662297 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:36.662308 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:36.662398 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:36.662515 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:36.673691 (+ 11176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:36.674804 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:36.676428 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:36.677011 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:36.677147 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:36.680507 (+ 3360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:36.680514 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:36.680519 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:36.680552 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:36.684190 (+ 3638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:36.685041 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:36.685046 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:36.685050 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:36.685233 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:36.686415 (+ 1182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:36.686422 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:36.686427 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:36.686699 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:36.686953 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:36.687276 (+ 323us) server_negotiation.cc:299] Negotiation successful
0925 15:13:36.687483 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31277,"thread_start_us":41193,"threads_started":1}
I20260925 15:13:36.688390 19720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:36.648564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51154 (local address 127.12.8.60:41857)
0925 15:13:36.659293 (+ 10729us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:36.659299 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:36.670642 (+ 11343us) server_negotiation.cc:407] Connection header received
0925 15:13:36.670941 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:36.670949 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:36.671061 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:36.671180 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:36.673628 (+ 2448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:36.674808 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:36.676760 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:36.677399 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:36.677527 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:36.681688 (+ 4161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:36.681695 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:36.681699 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:36.681744 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:36.685029 (+ 3285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:36.685838 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:36.685842 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:36.685844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:36.685967 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:36.686430 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:36.686435 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:36.686440 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:36.686774 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:36.687000 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:36.687553 (+ 553us) server_negotiation.cc:299] Negotiation successful
0925 15:13:36.687773 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10624,"thread_start_us":20187,"threads_started":1}
I20260925 15:13:36.687099 18460 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60696:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:13:36.694926 18460 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:13:36.721627 19740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:36.711565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.3:40103 (local address 127.0.0.1:51900)
0925 15:13:36.713355 (+ 1790us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:36.713374 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:36.713504 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:36.715052 (+ 1548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:36.715062 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:36.715089 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:13:36.715473 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:36.715486 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:36.716904 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:36.716912 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:36.718432 (+ 1520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:36.718445 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:36.721027 (+ 2582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:36.721032 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:36.721157 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:36.721182 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 15:13:36.721248 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 15:13:36.721346 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1671,"thread_start_us":2837,"threads_started":1}
I20260925 15:13:36.722069 19739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:36.712204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51900 (local address 127.12.8.3:40103)
0925 15:13:36.712884 (+ 680us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:36.712891 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:36.713660 (+ 769us) server_negotiation.cc:407] Connection header received
0925 15:13:36.713764 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:36.713771 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:36.714086 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:36.714341 (+ 255us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:13:36.715727 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:36.716769 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:36.718646 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:36.720835 (+ 2189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:36.721006 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:36.721238 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:36.721304 (+ 66us) server_negotiation.cc:299] Negotiation successful
0925 15:13:36.721465 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":931,"threads_started":1}
I20260925 15:13:36.726418 19646 tablet_service.cc:1467] Processing CreateTablet for tablet 092bd6d89c994ed39d40682aeb940646 (DEFAULT_TABLE table=test-table [id=f413ac818d1c425190a322a5dfabc9fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:13:36.728044 19646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 092bd6d89c994ed39d40682aeb940646. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:13:36.729735 19743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:36.715033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41384 (local address 127.12.8.2:35849)
0925 15:13:36.722791 (+ 7758us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:36.722800 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:36.722828 (+ 28us) server_negotiation.cc:407] Connection header received
0925 15:13:36.722923 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:36.722929 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:36.723196 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:36.723408 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:13:36.724175 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:36.725217 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:36.727489 (+ 2272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:36.728726 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:36.728892 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:36.729096 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:36.729308 (+ 212us) server_negotiation.cc:299] Negotiation successful
0925 15:13:36.729463 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7145,"thread_start_us":7021,"threads_started":1}
I20260925 15:13:36.729841 19742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:36.714782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.2:35849 (local address 127.0.0.1:41384)
0925 15:13:36.715398 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:36.715418 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:36.715541 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:36.723603 (+ 8062us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:36.723607 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:36.723627 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:13:36.724009 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:36.724019 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:36.725340 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:36.725348 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:36.727308 (+ 1960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:36.727323 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:36.728934 (+ 1611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:36.728941 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:36.729107 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:36.729154 (+ 47us) client_negotiation.cc:769] Sending connection context
0925 15:13:36.729270 (+ 116us) client_negotiation.cc:240] Negotiation successful
0925 15:13:36.729407 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"mutex_wait_us":149,"thread_start_us":247,"threads_started":1}
I20260925 15:13:36.730135 19738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:36.712803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.1:46055 (local address 127.0.0.1:49286)
0925 15:13:36.714711 (+ 1908us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:36.714728 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:36.714854 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:36.722392 (+ 7538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:36.722399 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:36.722432 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:13:36.722812 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:36.722823 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:36.725080 (+ 2257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:36.725085 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:36.727376 (+ 2291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:36.727394 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:36.729256 (+ 1862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:36.729266 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:36.729428 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:36.729459 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 15:13:36.729546 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 15:13:36.729683 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1784,"mutex_wait_us":102,"thread_start_us":556,"threads_started":1}
I20260925 15:13:36.731222 19741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:36.713369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49286 (local address 127.12.8.1:46055)
0925 15:13:36.714010 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:36.714022 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:36.721028 (+ 7006us) server_negotiation.cc:407] Connection header received
0925 15:13:36.721141 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:36.721149 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:36.721445 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:36.723063 (+ 1618us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:13:36.723345 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:36.724917 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:36.727597 (+ 2680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:36.728979 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:36.730493 (+ 1514us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:36.730716 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:36.730806 (+ 90us) server_negotiation.cc:299] Negotiation successful
0925 15:13:36.730964 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":272,"threads_started":1}
I20260925 15:13:36.733810 19505 tablet_service.cc:1467] Processing CreateTablet for tablet 092bd6d89c994ed39d40682aeb940646 (DEFAULT_TABLE table=test-table [id=f413ac818d1c425190a322a5dfabc9fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:13:36.734464 19361 tablet_service.cc:1467] Processing CreateTablet for tablet 092bd6d89c994ed39d40682aeb940646 (DEFAULT_TABLE table=test-table [id=f413ac818d1c425190a322a5dfabc9fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:13:36.735309 19505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 092bd6d89c994ed39d40682aeb940646. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:13:36.735620 19361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 092bd6d89c994ed39d40682aeb940646. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:13:36.743705 19744 tablet_bootstrap.cc:492] T 092bd6d89c994ed39d40682aeb940646 P 998c5926ac2741278b4a9c80cd3f8df5: Bootstrap starting.
I20260925 15:13:36.747367 19744 tablet_bootstrap.cc:654] T 092bd6d89c994ed39d40682aeb940646 P 998c5926ac2741278b4a9c80cd3f8df5: Neither blocks nor log segments found. Creating new log.
I20260925 15:13:36.748672 19745 tablet_bootstrap.cc:492] T 092bd6d89c994ed39d40682aeb940646 P 372093292e564acaae5d8d6370b2a26c: Bootstrap starting.
I20260925 15:13:36.748720 19744 log.cc:824] T 092bd6d89c994ed39d40682aeb940646 P 998c5926ac2741278b4a9c80cd3f8df5: Log is configured to *not* fsync() on all Append() calls
I20260925 15:13:36.753242 19745 tablet_bootstrap.cc:654] T 092bd6d89c994ed39d40682aeb940646 P 372093292e564acaae5d8d6370b2a26c: Neither blocks nor log segments found. Creating new log.
I20260925 15:13:36.754743 19745 log.cc:824] T 092bd6d89c994ed39d40682aeb940646 P 372093292e564acaae5d8d6370b2a26c: Log is configured to *not* fsync() on all Append() calls
I20260925 15:13:36.755717 19746 tablet_bootstrap.cc:492] T 092bd6d89c994ed39d40682aeb940646 P ce72ef070a95404f8dbb29d80d7cff69: Bootstrap starting.
I20260925 15:13:36.758663 19746 tablet_bootstrap.cc:654] T 092bd6d89c994ed39d40682aeb940646 P ce72ef070a95404f8dbb29d80d7cff69: Neither blocks nor log segments found. Creating new log.
I20260925 15:13:36.759294 19744 tablet_bootstrap.cc:492] T 092bd6d89c994ed39d40682aeb940646 P 998c5926ac2741278b4a9c80cd3f8df5: No bootstrap required, opened a new log
I20260925 15:13:36.759588 19744 ts_tablet_manager.cc:1397] T 092bd6d89c994ed39d40682aeb940646 P 998c5926ac2741278b4a9c80cd3f8df5: Time spent bootstrapping tablet: real 0.016s user 0.012s sys 0.001s
I20260925 15:13:36.759863 19746 log.cc:824] T 092bd6d89c994ed39d40682aeb940646 P ce72ef070a95404f8dbb29d80d7cff69: Log is configured to *not* fsync() on all Append() calls
I20260925 15:13:36.760564 19745 tablet_bootstrap.cc:492] T 092bd6d89c994ed39d40682aeb940646 P 372093292e564acaae5d8d6370b2a26c: No bootstrap required, opened a new log
I20260925 15:13:36.760828 19745 ts_tablet_manager.cc:1397] T 092bd6d89c994ed39d40682aeb940646 P 372093292e564acaae5d8d6370b2a26c: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.003s
I20260925 15:13:36.763118 19746 tablet_bootstrap.cc:492] T 092bd6d89c994ed39d40682aeb940646 P ce72ef070a95404f8dbb29d80d7cff69: No bootstrap required, opened a new log
I20260925 15:13:36.763351 19746 ts_tablet_manager.cc:1397] T 092bd6d89c994ed39d40682aeb940646 P ce72ef070a95404f8dbb29d80d7cff69: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260925 15:13:36.767987 19745 raft_consensus.cc:348] T 092bd6d89c994ed39d40682aeb940646 P 372093292e564acaae5d8d6370b2a26c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998c5926ac2741278b4a9c80cd3f8df5" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 40103 } } peers { permanent_uuid: "ce72ef070a95404f8dbb29d80d7cff69" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 35849 } } peers { permanent_uuid: "372093292e564acaae5d8d6370b2a26c" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 46055 } }
I20260925 15:13:36.768348 19745 raft_consensus.cc:374] T 092bd6d89c994ed39d40682aeb940646 P 372093292e564acaae5d8d6370b2a26c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:13:36.768451 19745 raft_consensus.cc:729] T 092bd6d89c994ed39d40682aeb940646 P 372093292e564acaae5d8d6370b2a26c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 372093292e564acaae5d8d6370b2a26c, State: Initialized, Role: FOLLOWER
I20260925 15:13:36.769093 19745 consensus_queue.cc:260] T 092bd6d89c994ed39d40682aeb940646 P 372093292e564acaae5d8d6370b2a26c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998c5926ac2741278b4a9c80cd3f8df5" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 40103 } } peers { permanent_uuid: "ce72ef070a95404f8dbb29d80d7cff69" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 35849 } } peers { permanent_uuid: "372093292e564acaae5d8d6370b2a26c" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 46055 } }
I20260925 15:13:36.770092 19744 raft_consensus.cc:348] T 092bd6d89c994ed39d40682aeb940646 P 998c5926ac2741278b4a9c80cd3f8df5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998c5926ac2741278b4a9c80cd3f8df5" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 40103 } } peers { permanent_uuid: "ce72ef070a95404f8dbb29d80d7cff69" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 35849 } } peers { permanent_uuid: "372093292e564acaae5d8d6370b2a26c" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 46055 } }
I20260925 15:13:36.770524 19744 raft_consensus.cc:374] T 092bd6d89c994ed39d40682aeb940646 P 998c5926ac2741278b4a9c80cd3f8df5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:13:36.770630 19744 raft_consensus.cc:729] T 092bd6d89c994ed39d40682aeb940646 P 998c5926ac2741278b4a9c80cd3f8df5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 998c5926ac2741278b4a9c80cd3f8df5, State: Initialized, Role: FOLLOWER
I20260925 15:13:36.771268 19744 consensus_queue.cc:260] T 092bd6d89c994ed39d40682aeb940646 P 998c5926ac2741278b4a9c80cd3f8df5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998c5926ac2741278b4a9c80cd3f8df5" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 40103 } } peers { permanent_uuid: "ce72ef070a95404f8dbb29d80d7cff69" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 35849 } } peers { permanent_uuid: "372093292e564acaae5d8d6370b2a26c" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 46055 } }
I20260925 15:13:36.771577 19745 ts_tablet_manager.cc:1428] T 092bd6d89c994ed39d40682aeb940646 P 372093292e564acaae5d8d6370b2a26c: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260925 15:13:36.774981 19746 raft_consensus.cc:348] T 092bd6d89c994ed39d40682aeb940646 P ce72ef070a95404f8dbb29d80d7cff69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998c5926ac2741278b4a9c80cd3f8df5" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 40103 } } peers { permanent_uuid: "ce72ef070a95404f8dbb29d80d7cff69" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 35849 } } peers { permanent_uuid: "372093292e564acaae5d8d6370b2a26c" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 46055 } }
I20260925 15:13:36.775470 19746 raft_consensus.cc:374] T 092bd6d89c994ed39d40682aeb940646 P ce72ef070a95404f8dbb29d80d7cff69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:13:36.775581 19746 raft_consensus.cc:729] T 092bd6d89c994ed39d40682aeb940646 P ce72ef070a95404f8dbb29d80d7cff69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce72ef070a95404f8dbb29d80d7cff69, State: Initialized, Role: FOLLOWER
I20260925 15:13:36.776348 19746 consensus_queue.cc:260] T 092bd6d89c994ed39d40682aeb940646 P ce72ef070a95404f8dbb29d80d7cff69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998c5926ac2741278b4a9c80cd3f8df5" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 40103 } } peers { permanent_uuid: "ce72ef070a95404f8dbb29d80d7cff69" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 35849 } } peers { permanent_uuid: "372093292e564acaae5d8d6370b2a26c" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 46055 } }
I20260925 15:13:36.780419 19744 ts_tablet_manager.cc:1428] T 092bd6d89c994ed39d40682aeb940646 P 998c5926ac2741278b4a9c80cd3f8df5: Time spent starting tablet: real 0.021s user 0.017s sys 0.003s
I20260925 15:13:36.781158 19712 heartbeater.cc:503] Master 127.12.8.62:44383 was elected leader, sending a full tablet report...
I20260925 15:13:36.781983 19571 heartbeater.cc:503] Master 127.12.8.62:44383 was elected leader, sending a full tablet report...
I20260925 15:13:36.784426 19746 ts_tablet_manager.cc:1428] T 092bd6d89c994ed39d40682aeb940646 P ce72ef070a95404f8dbb29d80d7cff69: Time spent starting tablet: real 0.021s user 0.015s sys 0.004s
W20260925 15:13:36.798396 19573 tablet.cc:2369] T 092bd6d89c994ed39d40682aeb940646 P ce72ef070a95404f8dbb29d80d7cff69: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:13:36.874157 19714 tablet.cc:2369] T 092bd6d89c994ed39d40682aeb940646 P 998c5926ac2741278b4a9c80cd3f8df5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:13:36.886971 19751 raft_consensus.cc:482] T 092bd6d89c994ed39d40682aeb940646 P 998c5926ac2741278b4a9c80cd3f8df5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:13:36.887292 19751 raft_consensus.cc:504] T 092bd6d89c994ed39d40682aeb940646 P 998c5926ac2741278b4a9c80cd3f8df5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998c5926ac2741278b4a9c80cd3f8df5" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 40103 } } peers { permanent_uuid: "ce72ef070a95404f8dbb29d80d7cff69" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 35849 } } peers { permanent_uuid: "372093292e564acaae5d8d6370b2a26c" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 46055 } }
I20260925 15:13:36.889019 19751 leader_election.cc:288] T 092bd6d89c994ed39d40682aeb940646 P 998c5926ac2741278b4a9c80cd3f8df5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce72ef070a95404f8dbb29d80d7cff69 (127.12.8.2:35849), 372093292e564acaae5d8d6370b2a26c (127.12.8.1:46055)
I20260925 15:13:36.894593 19743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:36.889341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.3:47011 (local address 127.12.8.2:35849)
0925 15:13:36.889592 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:36.889598 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:36.889788 (+ 190us) server_negotiation.cc:407] Connection header received
0925 15:13:36.889979 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:36.889983 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:36.890044 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:36.890138 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:13:36.890503 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:36.891221 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:36.892923 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:36.893671 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:36.893816 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:36.893905 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:36.894229 (+ 324us) server_negotiation.cc:299] Negotiation successful
0925 15:13:36.894348 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260925 15:13:36.894618 19719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:36.889264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.2:35849 (local address 127.12.8.3:47011)
0925 15:13:36.889672 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:36.889692 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:36.889836 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:36.890154 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:36.890158 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:36.890181 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:13:36.890371 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:36.890379 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:36.891355 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:36.891362 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:36.892773 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:36.892784 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:36.893829 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:36.893834 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:36.894044 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:36.894067 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 15:13:36.894178 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 15:13:36.894329 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"mutex_wait_us":226}
I20260925 15:13:36.894667 19716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:36.889263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.1:46055 (local address 127.12.8.3:55587)
0925 15:13:36.889438 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:36.889461 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:36.889596 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:36.890033 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:36.890037 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:36.890077 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:13:36.890300 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:36.890309 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:36.891328 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:36.891333 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:36.892785 (+ 1452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:36.892797 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:36.893874 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:36.893880 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:36.894089 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:36.894115 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 15:13:36.894254 (+ 139us) client_negotiation.cc:240] Negotiation successful
0925 15:13:36.894380 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58,"mutex_wait_us":81}
I20260925 15:13:36.894656 19741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:36.889444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.3:55587 (local address 127.12.8.1:46055)
0925 15:13:36.889769 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:36.889773 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:36.889787 (+ 14us) server_negotiation.cc:407] Connection header received
0925 15:13:36.889844 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:36.889847 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:36.889903 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:36.890002 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:13:36.890454 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:36.891183 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:36.892948 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:36.893704 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:36.893853 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:36.893940 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:36.894284 (+ 344us) server_negotiation.cc:299] Negotiation successful
0925 15:13:36.894402 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191}
I20260925 15:13:36.895612 19381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "092bd6d89c994ed39d40682aeb940646" candidate_uuid: "998c5926ac2741278b4a9c80cd3f8df5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "372093292e564acaae5d8d6370b2a26c" is_pre_election: true
I20260925 15:13:36.895615 19525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "092bd6d89c994ed39d40682aeb940646" candidate_uuid: "998c5926ac2741278b4a9c80cd3f8df5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce72ef070a95404f8dbb29d80d7cff69" is_pre_election: true
I20260925 15:13:36.896095 19525 raft_consensus.cc:2454] T 092bd6d89c994ed39d40682aeb940646 P ce72ef070a95404f8dbb29d80d7cff69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 998c5926ac2741278b4a9c80cd3f8df5 in term 0.
I20260925 15:13:36.896073 19381 raft_consensus.cc:2454] T 092bd6d89c994ed39d40682aeb940646 P 372093292e564acaae5d8d6370b2a26c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 998c5926ac2741278b4a9c80cd3f8df5 in term 0.
I20260925 15:13:36.896719 19599 leader_election.cc:302] T 092bd6d89c994ed39d40682aeb940646 P 998c5926ac2741278b4a9c80cd3f8df5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 372093292e564acaae5d8d6370b2a26c, 998c5926ac2741278b4a9c80cd3f8df5, ce72ef070a95404f8dbb29d80d7cff69; no voters:
I20260925 15:13:36.897112 19751 raft_consensus.cc:2781] T 092bd6d89c994ed39d40682aeb940646 P 998c5926ac2741278b4a9c80cd3f8df5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:13:36.897257 19751 raft_consensus.cc:482] T 092bd6d89c994ed39d40682aeb940646 P 998c5926ac2741278b4a9c80cd3f8df5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:13:36.897360 19751 raft_consensus.cc:3037] T 092bd6d89c994ed39d40682aeb940646 P 998c5926ac2741278b4a9c80cd3f8df5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:13:36.899668 19751 raft_consensus.cc:504] T 092bd6d89c994ed39d40682aeb940646 P 998c5926ac2741278b4a9c80cd3f8df5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998c5926ac2741278b4a9c80cd3f8df5" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 40103 } } peers { permanent_uuid: "ce72ef070a95404f8dbb29d80d7cff69" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 35849 } } peers { permanent_uuid: "372093292e564acaae5d8d6370b2a26c" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 46055 } }
I20260925 15:13:36.900250 19751 leader_election.cc:288] T 092bd6d89c994ed39d40682aeb940646 P 998c5926ac2741278b4a9c80cd3f8df5 [CANDIDATE]: Term 1 election: Requested vote from peers ce72ef070a95404f8dbb29d80d7cff69 (127.12.8.2:35849), 372093292e564acaae5d8d6370b2a26c (127.12.8.1:46055)
I20260925 15:13:36.900586 19525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "092bd6d89c994ed39d40682aeb940646" candidate_uuid: "998c5926ac2741278b4a9c80cd3f8df5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce72ef070a95404f8dbb29d80d7cff69"
I20260925 15:13:36.900779 19525 raft_consensus.cc:3037] T 092bd6d89c994ed39d40682aeb940646 P ce72ef070a95404f8dbb29d80d7cff69 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:13:36.900769 19381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "092bd6d89c994ed39d40682aeb940646" candidate_uuid: "998c5926ac2741278b4a9c80cd3f8df5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "372093292e564acaae5d8d6370b2a26c"
I20260925 15:13:36.900954 19381 raft_consensus.cc:3037] T 092bd6d89c994ed39d40682aeb940646 P 372093292e564acaae5d8d6370b2a26c [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:13:36.903029 19525 raft_consensus.cc:2454] T 092bd6d89c994ed39d40682aeb940646 P ce72ef070a95404f8dbb29d80d7cff69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 998c5926ac2741278b4a9c80cd3f8df5 in term 1.
I20260925 15:13:36.903154 19381 raft_consensus.cc:2454] T 092bd6d89c994ed39d40682aeb940646 P 372093292e564acaae5d8d6370b2a26c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 998c5926ac2741278b4a9c80cd3f8df5 in term 1.
I20260925 15:13:36.903496 19601 leader_election.cc:302] T 092bd6d89c994ed39d40682aeb940646 P 998c5926ac2741278b4a9c80cd3f8df5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 998c5926ac2741278b4a9c80cd3f8df5, ce72ef070a95404f8dbb29d80d7cff69; no voters:
I20260925 15:13:36.903947 19751 raft_consensus.cc:2781] T 092bd6d89c994ed39d40682aeb940646 P 998c5926ac2741278b4a9c80cd3f8df5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:13:36.904424 19751 raft_consensus.cc:686] T 092bd6d89c994ed39d40682aeb940646 P 998c5926ac2741278b4a9c80cd3f8df5 [term 1 LEADER]: Becoming Leader. State: Replica: 998c5926ac2741278b4a9c80cd3f8df5, State: Running, Role: LEADER
I20260925 15:13:36.904965 19751 consensus_queue.cc:237] T 092bd6d89c994ed39d40682aeb940646 P 998c5926ac2741278b4a9c80cd3f8df5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998c5926ac2741278b4a9c80cd3f8df5" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 40103 } } peers { permanent_uuid: "ce72ef070a95404f8dbb29d80d7cff69" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 35849 } } peers { permanent_uuid: "372093292e564acaae5d8d6370b2a26c" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 46055 } }
I20260925 15:13:36.909850 18460 catalog_manager.cc:5495] T 092bd6d89c994ed39d40682aeb940646 P 998c5926ac2741278b4a9c80cd3f8df5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 998c5926ac2741278b4a9c80cd3f8df5 (127.12.8.3). New cstate: current_term: 1 leader_uuid: "998c5926ac2741278b4a9c80cd3f8df5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998c5926ac2741278b4a9c80cd3f8df5" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 40103 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce72ef070a95404f8dbb29d80d7cff69" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 35849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "372093292e564acaae5d8d6370b2a26c" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 46055 } health_report { overall_health: UNKNOWN } } }
I20260925 15:13:36.961865 19739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:36.956583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51902 (local address 127.12.8.3:40103)
0925 15:13:36.956755 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:36.956759 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:36.956878 (+ 119us) server_negotiation.cc:407] Connection header received
0925 15:13:36.957047 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:36.957050 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:36.957136 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:36.957265 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:13:36.957618 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:36.958393 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:36.959704 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:36.960112 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:36.960250 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:36.960732 (+ 482us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:13:36.961259 (+ 527us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:13:36.961423 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:36.961532 (+ 109us) server_negotiation.cc:299] Negotiation successful
0925 15:13:36.961610 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260925 15:13:36.968775 19429 tablet.cc:2369] T 092bd6d89c994ed39d40682aeb940646 P 372093292e564acaae5d8d6370b2a26c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:13:36.969584 19381 raft_consensus.cc:1261] T 092bd6d89c994ed39d40682aeb940646 P 372093292e564acaae5d8d6370b2a26c [term 1 FOLLOWER]: Refusing update from remote peer 998c5926ac2741278b4a9c80cd3f8df5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:13:36.969714 19525 raft_consensus.cc:1261] T 092bd6d89c994ed39d40682aeb940646 P ce72ef070a95404f8dbb29d80d7cff69 [term 1 FOLLOWER]: Refusing update from remote peer 998c5926ac2741278b4a9c80cd3f8df5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:13:36.970710 19753 consensus_queue.cc:1035] T 092bd6d89c994ed39d40682aeb940646 P 998c5926ac2741278b4a9c80cd3f8df5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce72ef070a95404f8dbb29d80d7cff69" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 35849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:13:36.971084 19751 consensus_queue.cc:1035] T 092bd6d89c994ed39d40682aeb940646 P 998c5926ac2741278b4a9c80cd3f8df5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "372093292e564acaae5d8d6370b2a26c" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 46055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:13:36.986289 19758 mvcc.cc:205] Tried to move back new op lower bound from 7333270392698011648 to 7333270392446509056. Current Snapshot: MvccSnapshot[applied={T|T < 7333270392698011648}]
I20260925 15:13:36.988591 19756 mvcc.cc:205] Tried to move back new op lower bound from 7333270392698011648 to 7333270392446509056. Current Snapshot: MvccSnapshot[applied={T|T < 7333270392698011648 or (T in {7333270392698011648})}]
I20260925 15:13:36.989761 19757 mvcc.cc:205] Tried to move back new op lower bound from 7333270392698011648 to 7333270392446509056. Current Snapshot: MvccSnapshot[applied={T|T < 7333270392698011648}]
I20260925 15:13:37.001879 18460 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60696:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:13:37.002151 18460 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60696:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:13:37.008174 18460 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c1233ca1ad3745eca6025d08fc59d7f3: Sending DeleteTablet for 3 replicas of tablet 092bd6d89c994ed39d40682aeb940646
I20260925 15:13:37.009181 19646 tablet_service.cc:1514] Processing DeleteTablet for tablet 092bd6d89c994ed39d40682aeb940646 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:13:36 UTC) from {username='oryx', principal='oryx/127.12.8.62@KRBTEST.COM'} at 127.0.0.1:51900
I20260925 15:13:37.009272 19505 tablet_service.cc:1514] Processing DeleteTablet for tablet 092bd6d89c994ed39d40682aeb940646 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:13:36 UTC) from {username='oryx', principal='oryx/127.12.8.62@KRBTEST.COM'} at 127.0.0.1:41384
I20260925 15:13:37.010254 19765 tablet_replica.cc:330] T 092bd6d89c994ed39d40682aeb940646 P 998c5926ac2741278b4a9c80cd3f8df5: stopping tablet replica
I20260925 15:13:37.010430 19361 tablet_service.cc:1514] Processing DeleteTablet for tablet 092bd6d89c994ed39d40682aeb940646 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:13:36 UTC) from {username='oryx', principal='oryx/127.12.8.62@KRBTEST.COM'} at 127.0.0.1:49286
I20260925 15:13:37.010946 19765 raft_consensus.cc:2229] T 092bd6d89c994ed39d40682aeb940646 P 998c5926ac2741278b4a9c80cd3f8df5 [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:13:37.011826 19765 raft_consensus.cc:2258] T 092bd6d89c994ed39d40682aeb940646 P 998c5926ac2741278b4a9c80cd3f8df5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:13:37.014309 19765 ts_tablet_manager.cc:1905] T 092bd6d89c994ed39d40682aeb940646 P 998c5926ac2741278b4a9c80cd3f8df5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:13:37.022666 19767 tablet_replica.cc:330] T 092bd6d89c994ed39d40682aeb940646 P 372093292e564acaae5d8d6370b2a26c: stopping tablet replica
I20260925 15:13:37.023572 19765 ts_tablet_manager.cc:1918] T 092bd6d89c994ed39d40682aeb940646 P 998c5926ac2741278b4a9c80cd3f8df5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:13:37.023862 19765 log.cc:1196] T 092bd6d89c994ed39d40682aeb940646 P 998c5926ac2741278b4a9c80cd3f8df5: Deleting WAL directory at /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/ts-2/wal/wals/092bd6d89c994ed39d40682aeb940646
I20260925 15:13:37.023986 19767 raft_consensus.cc:2229] T 092bd6d89c994ed39d40682aeb940646 P 372093292e564acaae5d8d6370b2a26c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:13:37.024212 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 18428
I20260925 15:13:37.024513 19765 ts_tablet_manager.cc:1939] T 092bd6d89c994ed39d40682aeb940646 P 998c5926ac2741278b4a9c80cd3f8df5: Deleting consensus metadata
I20260925 15:13:37.025341 18490 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:13:37.024557 19767 raft_consensus.cc:2258] T 092bd6d89c994ed39d40682aeb940646 P 372093292e564acaae5d8d6370b2a26c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:13:37.027980 19766 tablet_replica.cc:330] T 092bd6d89c994ed39d40682aeb940646 P ce72ef070a95404f8dbb29d80d7cff69: stopping tablet replica
I20260925 15:13:37.028489 19766 raft_consensus.cc:2229] T 092bd6d89c994ed39d40682aeb940646 P ce72ef070a95404f8dbb29d80d7cff69 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:13:37.028950 19766 raft_consensus.cc:2258] T 092bd6d89c994ed39d40682aeb940646 P ce72ef070a95404f8dbb29d80d7cff69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:13:37.032163 19767 ts_tablet_manager.cc:1905] T 092bd6d89c994ed39d40682aeb940646 P 372093292e564acaae5d8d6370b2a26c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:13:37.034354 19766 ts_tablet_manager.cc:1905] T 092bd6d89c994ed39d40682aeb940646 P ce72ef070a95404f8dbb29d80d7cff69: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:13:37.038082 19767 ts_tablet_manager.cc:1918] T 092bd6d89c994ed39d40682aeb940646 P 372093292e564acaae5d8d6370b2a26c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:13:37.038275 19767 log.cc:1196] T 092bd6d89c994ed39d40682aeb940646 P 372093292e564acaae5d8d6370b2a26c: Deleting WAL directory at /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/ts-0/wal/wals/092bd6d89c994ed39d40682aeb940646
I20260925 15:13:37.038837 19767 ts_tablet_manager.cc:1939] T 092bd6d89c994ed39d40682aeb940646 P 372093292e564acaae5d8d6370b2a26c: Deleting consensus metadata
I20260925 15:13:37.043740 19766 ts_tablet_manager.cc:1918] T 092bd6d89c994ed39d40682aeb940646 P ce72ef070a95404f8dbb29d80d7cff69: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:13:37.043913 19766 log.cc:1196] T 092bd6d89c994ed39d40682aeb940646 P ce72ef070a95404f8dbb29d80d7cff69: Deleting WAL directory at /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/ts-1/wal/wals/092bd6d89c994ed39d40682aeb940646
I20260925 15:13:37.044539 19766 ts_tablet_manager.cc:1939] T 092bd6d89c994ed39d40682aeb940646 P ce72ef070a95404f8dbb29d80d7cff69: Deleting consensus metadata
I20260925 15:13:37.045655 19715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:37.014772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49522 (local address 127.12.8.61:35001)
0925 15:13:37.014971 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:37.014979 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:37.022931 (+ 7952us) server_negotiation.cc:407] Connection header received
0925 15:13:37.023184 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:37.023191 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:37.023284 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:37.023400 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:37.026907 (+ 3507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:37.028335 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:37.030245 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:37.030867 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:37.030994 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:37.038935 (+ 7941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:37.038941 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:37.038948 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:37.038995 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:37.042541 (+ 3546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:37.043330 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:37.043335 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:37.043339 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:37.043494 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:37.043941 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:37.043946 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:37.043949 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:37.044473 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:37.044779 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:37.045108 (+ 329us) server_negotiation.cc:299] Negotiation successful
0925 15:13:37.045314 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 15:13:37.060143 19720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:37.023764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51156 (local address 127.12.8.60:41857)
0925 15:13:37.023957 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:37.023963 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:37.045911 (+ 21948us) server_negotiation.cc:407] Connection header received
0925 15:13:37.045995 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:37.046000 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:37.046088 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:37.046181 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:37.047906 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:37.048634 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:37.049962 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:37.050473 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:37.050604 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:37.053015 (+ 2411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:37.053021 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:37.053026 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:37.053068 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:37.056429 (+ 3361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:37.057261 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:37.057266 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:37.057269 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:37.057426 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:37.057854 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:37.057859 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:37.057862 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:37.058251 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:37.058498 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:37.059567 (+ 1069us) server_negotiation.cc:299] Negotiation successful
0925 15:13:37.059773 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 15:13:37.170881 18444 catalog_manager.cc:4841] TS 372093292e564acaae5d8d6370b2a26c (127.12.8.1:46055): tablet 092bd6d89c994ed39d40682aeb940646 (table test-table [id=f413ac818d1c425190a322a5dfabc9fe]) successfully deleted
I20260925 15:13:37.172812 18446 catalog_manager.cc:4841] TS 998c5926ac2741278b4a9c80cd3f8df5 (127.12.8.3:40103): tablet 092bd6d89c994ed39d40682aeb940646 (table test-table [id=f413ac818d1c425190a322a5dfabc9fe]) successfully deleted
I20260925 15:13:37.172901 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 18428
W20260925 15:13:37.194757 19461 connection.cc:535] server connection from 127.0.0.1:41384 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 15:13:37.215900 19720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:37.200151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51160 (local address 127.12.8.60:41857)
0925 15:13:37.200345 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:37.200351 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:37.201015 (+ 664us) server_negotiation.cc:407] Connection header received
0925 15:13:37.201189 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:37.201195 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:37.201273 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:37.201382 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:37.203406 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:37.204259 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:37.205665 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:37.206358 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:37.206492 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:37.209825 (+ 3333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:37.209831 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:37.209838 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:37.209880 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:37.213025 (+ 3145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:37.213778 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:37.213782 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:37.213784 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:37.213893 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:37.214311 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:37.214318 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:37.214321 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:37.214722 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:37.214977 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:37.215343 (+ 366us) server_negotiation.cc:299] Negotiation successful
0925 15:13:37.215555 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 15:13:37.216702 19715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:37.199104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49526 (local address 127.12.8.61:35001)
0925 15:13:37.199319 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:37.199326 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:37.200177 (+ 851us) server_negotiation.cc:407] Connection header received
0925 15:13:37.200621 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:37.200630 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:37.200729 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:37.200838 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:37.202895 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:37.203796 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:37.205780 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:37.206492 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:37.206601 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:37.209321 (+ 2720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:37.209327 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:37.209332 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:37.209375 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:37.213025 (+ 3650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:37.214441 (+ 1416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:37.214447 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:37.214451 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:37.214588 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:37.215028 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:37.215034 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:37.215037 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:37.215379 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:37.215590 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:37.216178 (+ 588us) server_negotiation.cc:299] Negotiation successful
0925 15:13:37.216389 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 15:13:37.991343 19776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:37.990343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:44383 (local address 127.12.8.3:55965)
0925 15:13:37.990881 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:37.991061 (+ 180us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:44383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":259,"threads_started":1}
W20260925 15:13:37.991968 19712 heartbeater.cc:644] Failed to heartbeat to 127.12.8.62:44383 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.12.8.62:44383: connect: Connection refused (error 111)
I20260925 15:13:37.993834 19777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:37.993031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:44383 (local address 127.12.8.1:43627)
0925 15:13:37.993513 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:37.993611 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:44383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":274,"threads_started":1}
W20260925 15:13:37.994349 19427 heartbeater.cc:644] Failed to heartbeat to 127.12.8.62:44383 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.12.8.62:44383: connect: Connection refused (error 111)
I20260925 15:13:37.999030 19778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:37.998400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:44383 (local address 127.12.8.2:53553)
0925 15:13:37.998756 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:37.998848 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:44383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":190,"threads_started":1}
W20260925 15:13:37.999418 19571 heartbeater.cc:644] Failed to heartbeat to 127.12.8.62:44383 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.12.8.62:44383: connect: Connection refused (error 111)
I20260925 15:13:38.508548 19779 raft_consensus.cc:482] T 00000000000000000000000000000000 P 01c12756ade6445f92d995ab935f60d0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c1233ca1ad3745eca6025d08fc59d7f3)
I20260925 15:13:38.508841 19779 raft_consensus.cc:504] T 00000000000000000000000000000000 P 01c12756ade6445f92d995ab935f60d0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c1233ca1ad3745eca6025d08fc59d7f3" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 44383 } } peers { permanent_uuid: "ea8b382304f143f4b4a3dca87c09b7e4" member_type: VOTER last_known_addr { host: "127.12.8.61" port: 35001 } attrs { promote: false } } peers { permanent_uuid: "01c12756ade6445f92d995ab935f60d0" member_type: VOTER last_known_addr { host: "127.12.8.60" port: 41857 } attrs { promote: false } }
I20260925 15:13:38.510250 19779 leader_election.cc:288] T 00000000000000000000000000000000 P 01c12756ade6445f92d995ab935f60d0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1233ca1ad3745eca6025d08fc59d7f3 (127.12.8.62:44383), ea8b382304f143f4b4a3dca87c09b7e4 (127.12.8.61:35001)
I20260925 15:13:38.511557 19780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:38.510455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:44383 (local address 127.0.0.1:32898)
0925 15:13:38.511045 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:38.511244 (+ 199us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:44383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":263,"threads_started":1}
W20260925 15:13:38.511814 19235 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.8.62:44383: connect: Connection refused (error 111)
I20260925 15:13:38.512423 19783 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea8b382304f143f4b4a3dca87c09b7e4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c1233ca1ad3745eca6025d08fc59d7f3)
I20260925 15:13:38.512833 19783 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea8b382304f143f4b4a3dca87c09b7e4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c1233ca1ad3745eca6025d08fc59d7f3" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 44383 } } peers { permanent_uuid: "ea8b382304f143f4b4a3dca87c09b7e4" member_type: VOTER last_known_addr { host: "127.12.8.61" port: 35001 } attrs { promote: false } } peers { permanent_uuid: "01c12756ade6445f92d995ab935f60d0" member_type: VOTER last_known_addr { host: "127.12.8.60" port: 41857 } attrs { promote: false } }
I20260925 15:13:38.513758 19780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:38.513397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:44383 (local address 127.0.0.1:32904)
0925 15:13:38.513555 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:38.513635 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:44383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260925 15:13:38.513928 19235 leader_election.cc:334] T 00000000000000000000000000000000 P 01c12756ade6445f92d995ab935f60d0 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1233ca1ad3745eca6025d08fc59d7f3 (127.12.8.62:44383): Network error: Client connection negotiation failed: client connection to 127.12.8.62:44383: connect: Connection refused (error 111)
I20260925 15:13:38.514379 19783 leader_election.cc:288] T 00000000000000000000000000000000 P ea8b382304f143f4b4a3dca87c09b7e4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1233ca1ad3745eca6025d08fc59d7f3 (127.12.8.62:44383), 01c12756ade6445f92d995ab935f60d0 (127.12.8.60:41857)
I20260925 15:13:38.522313 19782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:38.511327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.61:35001 (local address 127.0.0.1:49542)
0925 15:13:38.512218 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:38.512238 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:38.512406 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:38.513019 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:38.513023 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:38.513043 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:13:38.513399 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:38.513411 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:38.515602 (+ 2191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:38.515611 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:38.518810 (+ 3199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:38.518831 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:38.521167 (+ 2336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:38.521176 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:38.521450 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:38.521516 (+ 66us) client_negotiation.cc:769] Sending connection context
0925 15:13:38.521624 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 15:13:38.521831 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":762,"thread_start_us":984,"threads_started":1}
I20260925 15:13:38.523288 19781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:38.510584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49542 (local address 127.12.8.61:35001)
0925 15:13:38.511965 (+ 1381us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:38.511974 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:38.512557 (+ 583us) server_negotiation.cc:407] Connection header received
0925 15:13:38.512664 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:38.512670 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:38.512740 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:38.512844 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:13:38.513596 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:38.515365 (+ 1769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:38.519027 (+ 3662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:38.520717 (+ 1690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:38.520894 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:38.521040 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:38.522584 (+ 1544us) server_negotiation.cc:299] Negotiation successful
0925 15:13:38.522772 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1241,"thread_start_us":1096,"threads_started":1}
I20260925 15:13:38.524493 19068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "01c12756ade6445f92d995ab935f60d0" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ea8b382304f143f4b4a3dca87c09b7e4" is_pre_election: true
I20260925 15:13:38.525038 19786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:38.514448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:44383 (local address 127.0.0.1:32918)
0925 15:13:38.523974 (+ 9526us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:38.524198 (+ 224us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:44383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":9371,"threads_started":1}
I20260925 15:13:38.525096 19068 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ea8b382304f143f4b4a3dca87c09b7e4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01c12756ade6445f92d995ab935f60d0 in term 1.
I20260925 15:13:38.525662 19232 leader_election.cc:302] T 00000000000000000000000000000000 P 01c12756ade6445f92d995ab935f60d0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01c12756ade6445f92d995ab935f60d0, ea8b382304f143f4b4a3dca87c09b7e4; no voters: c1233ca1ad3745eca6025d08fc59d7f3
W20260925 15:13:38.525761 19045 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.8.62:44383: connect: Connection refused (error 111)
I20260925 15:13:38.526157 19779 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 01c12756ade6445f92d995ab935f60d0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 15:13:38.526326 19779 raft_consensus.cc:482] T 00000000000000000000000000000000 P 01c12756ade6445f92d995ab935f60d0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader c1233ca1ad3745eca6025d08fc59d7f3)
I20260925 15:13:38.526448 19779 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 01c12756ade6445f92d995ab935f60d0 [term 1 FOLLOWER]: Advancing to term 2
I20260925 15:13:38.528239 19785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:38.527546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:44383 (local address 127.0.0.1:32926)
0925 15:13:38.527750 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:38.527910 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:44383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
W20260925 15:13:38.528451 19045 leader_election.cc:334] T 00000000000000000000000000000000 P ea8b382304f143f4b4a3dca87c09b7e4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1233ca1ad3745eca6025d08fc59d7f3 (127.12.8.62:44383): Network error: Client connection negotiation failed: client connection to 127.12.8.62:44383: connect: Connection refused (error 111)
I20260925 15:13:38.529311 19779 raft_consensus.cc:504] T 00000000000000000000000000000000 P 01c12756ade6445f92d995ab935f60d0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c1233ca1ad3745eca6025d08fc59d7f3" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 44383 } } peers { permanent_uuid: "ea8b382304f143f4b4a3dca87c09b7e4" member_type: VOTER last_known_addr { host: "127.12.8.61" port: 35001 } attrs { promote: false } } peers { permanent_uuid: "01c12756ade6445f92d995ab935f60d0" member_type: VOTER last_known_addr { host: "127.12.8.60" port: 41857 } attrs { promote: false } }
I20260925 15:13:38.530035 19779 leader_election.cc:288] T 00000000000000000000000000000000 P 01c12756ade6445f92d995ab935f60d0 [CANDIDATE]: Term 2 election: Requested vote from peers c1233ca1ad3745eca6025d08fc59d7f3 (127.12.8.62:44383), ea8b382304f143f4b4a3dca87c09b7e4 (127.12.8.61:35001)
I20260925 15:13:38.530575 19782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:38.530174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:44383 (local address 127.0.0.1:32940)
0925 15:13:38.530347 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:38.530439 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:44383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260925 15:13:38.530576 19068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "01c12756ade6445f92d995ab935f60d0" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ea8b382304f143f4b4a3dca87c09b7e4"
I20260925 15:13:38.530881 19068 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea8b382304f143f4b4a3dca87c09b7e4 [term 1 FOLLOWER]: Advancing to term 2
I20260925 15:13:38.531855 19782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:38.531403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:44383 (local address 127.0.0.1:32944)
0925 15:13:38.531594 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:38.531710 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:44383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
W20260925 15:13:38.532068 19235 leader_election.cc:334] T 00000000000000000000000000000000 P 01c12756ade6445f92d995ab935f60d0 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer c1233ca1ad3745eca6025d08fc59d7f3 (127.12.8.62:44383): Network error: Client connection negotiation failed: client connection to 127.12.8.62:44383: connect: Connection refused (error 111)
I20260925 15:13:38.532501 19787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:38.514873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51164 (local address 127.12.8.60:41857)
0925 15:13:38.524984 (+ 10111us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:38.524990 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:38.525580 (+ 590us) server_negotiation.cc:407] Connection header received
0925 15:13:38.525639 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:38.525644 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:38.525710 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:38.525819 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:13:38.526746 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:38.527981 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:38.529833 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:38.531374 (+ 1541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:38.531518 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:38.531630 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:38.532041 (+ 411us) server_negotiation.cc:299] Negotiation successful
0925 15:13:38.532171 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9959,"thread_start_us":9820,"threads_started":1}
I20260925 15:13:38.532536 19786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:38.514553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.60:41857 (local address 127.0.0.1:51164)
0925 15:13:38.525278 (+ 10725us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:38.525304 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:38.525448 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:38.526001 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:38.526006 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:38.526042 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:13:38.526476 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:38.526496 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:38.528124 (+ 1628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:38.528131 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:38.529680 (+ 1549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:38.529692 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:38.531567 (+ 1875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:38.531573 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:38.531804 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:38.531828 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 15:13:38.532021 (+ 193us) client_negotiation.cc:240] Negotiation successful
0925 15:13:38.532183 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10603,"thread_start_us":9130,"threads_started":1}
I20260925 15:13:38.533278 19258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ea8b382304f143f4b4a3dca87c09b7e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "01c12756ade6445f92d995ab935f60d0" is_pre_election: true
I20260925 15:13:38.533610 19258 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 01c12756ade6445f92d995ab935f60d0 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ea8b382304f143f4b4a3dca87c09b7e4 in current term 2: Already voted for candidate 01c12756ade6445f92d995ab935f60d0 in this term.
I20260925 15:13:38.533993 19068 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ea8b382304f143f4b4a3dca87c09b7e4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01c12756ade6445f92d995ab935f60d0 in term 2.
I20260925 15:13:38.534173 19042 leader_election.cc:302] T 00000000000000000000000000000000 P ea8b382304f143f4b4a3dca87c09b7e4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea8b382304f143f4b4a3dca87c09b7e4; no voters: 01c12756ade6445f92d995ab935f60d0, c1233ca1ad3745eca6025d08fc59d7f3
I20260925 15:13:38.534497 19783 raft_consensus.cc:2726] T 00000000000000000000000000000000 P ea8b382304f143f4b4a3dca87c09b7e4 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 15:13:38.534844 19232 leader_election.cc:302] T 00000000000000000000000000000000 P 01c12756ade6445f92d995ab935f60d0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01c12756ade6445f92d995ab935f60d0, ea8b382304f143f4b4a3dca87c09b7e4; no voters: c1233ca1ad3745eca6025d08fc59d7f3
I20260925 15:13:38.535108 19779 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 01c12756ade6445f92d995ab935f60d0 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 15:13:38.535437 19779 raft_consensus.cc:686] T 00000000000000000000000000000000 P 01c12756ade6445f92d995ab935f60d0 [term 2 LEADER]: Becoming Leader. State: Replica: 01c12756ade6445f92d995ab935f60d0, State: Running, Role: LEADER
I20260925 15:13:38.535831 19779 consensus_queue.cc:237] T 00000000000000000000000000000000 P 01c12756ade6445f92d995ab935f60d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c1233ca1ad3745eca6025d08fc59d7f3" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 44383 } } peers { permanent_uuid: "ea8b382304f143f4b4a3dca87c09b7e4" member_type: VOTER last_known_addr { host: "127.12.8.61" port: 35001 } attrs { promote: false } } peers { permanent_uuid: "01c12756ade6445f92d995ab935f60d0" member_type: VOTER last_known_addr { host: "127.12.8.60" port: 41857 } attrs { promote: false } }
I20260925 15:13:38.537773 19789 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01c12756ade6445f92d995ab935f60d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 01c12756ade6445f92d995ab935f60d0. Latest consensus state: current_term: 2 leader_uuid: "01c12756ade6445f92d995ab935f60d0" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c1233ca1ad3745eca6025d08fc59d7f3" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 44383 } } peers { permanent_uuid: "ea8b382304f143f4b4a3dca87c09b7e4" member_type: VOTER last_known_addr { host: "127.12.8.61" port: 35001 } attrs { promote: false } } peers { permanent_uuid: "01c12756ade6445f92d995ab935f60d0" member_type: VOTER last_known_addr { host: "127.12.8.60" port: 41857 } attrs { promote: false } } }
I20260925 15:13:38.538152 19789 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01c12756ade6445f92d995ab935f60d0 [sys.catalog]: This master's current role is: LEADER
I20260925 15:13:38.539078 19791 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:13:38.542665 19791 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:13:38.543385 19791 catalog_manager.cc:1259] Loaded cluster ID: 5911822a4ce34211bb32228d45a52d8f
I20260925 15:13:38.543452 19791 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:13:38.544435 19791 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:13:38.545258 19791 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 01c12756ade6445f92d995ab935f60d0: Loaded TSK: 0
I20260925 15:13:38.546265 19791 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:13:38.594599 19247 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51160:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:13:38.601617 19247 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:13:38.605719 19782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:38.605307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:44383 (local address 127.0.0.1:32976)
0925 15:13:38.605470 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:38.605562 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:44383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260925 15:13:38.605849 19068 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ea8b382304f143f4b4a3dca87c09b7e4 [term 2 FOLLOWER]: Refusing update from remote peer 01c12756ade6445f92d995ab935f60d0: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260925 15:13:38.606829 19782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:38.606386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:44383 (local address 127.0.0.1:32992)
0925 15:13:38.606577 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:38.606667 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:44383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260925 15:13:38.606748 19789 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 01c12756ade6445f92d995ab935f60d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea8b382304f143f4b4a3dca87c09b7e4" member_type: VOTER last_known_addr { host: "127.12.8.61" port: 35001 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260925 15:13:38.607050 19235 consensus_peers.cc:487] T 00000000000000000000000000000000 P 01c12756ade6445f92d995ab935f60d0 -> Peer c1233ca1ad3745eca6025d08fc59d7f3 (127.12.8.62:44383): Couldn't send request to peer c1233ca1ad3745eca6025d08fc59d7f3. Status: Network error: Client connection negotiation failed: client connection to 127.12.8.62:44383: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 15:13:38.609493 19783 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea8b382304f143f4b4a3dca87c09b7e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 01c12756ade6445f92d995ab935f60d0. Latest consensus state: current_term: 2 leader_uuid: "01c12756ade6445f92d995ab935f60d0" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c1233ca1ad3745eca6025d08fc59d7f3" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 44383 } } peers { permanent_uuid: "ea8b382304f143f4b4a3dca87c09b7e4" member_type: VOTER last_known_addr { host: "127.12.8.61" port: 35001 } attrs { promote: false } } peers { permanent_uuid: "01c12756ade6445f92d995ab935f60d0" member_type: VOTER last_known_addr { host: "127.12.8.60" port: 41857 } attrs { promote: false } } }
I20260925 15:13:38.609725 19783 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea8b382304f143f4b4a3dca87c09b7e4 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:13:38.611191 19779 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01c12756ade6445f92d995ab935f60d0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "01c12756ade6445f92d995ab935f60d0" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c1233ca1ad3745eca6025d08fc59d7f3" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 44383 } } peers { permanent_uuid: "ea8b382304f143f4b4a3dca87c09b7e4" member_type: VOTER last_known_addr { host: "127.12.8.61" port: 35001 } attrs { promote: false } } peers { permanent_uuid: "01c12756ade6445f92d995ab935f60d0" member_type: VOTER last_known_addr { host: "127.12.8.60" port: 41857 } attrs { promote: false } } }
I20260925 15:13:38.611497 19779 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01c12756ade6445f92d995ab935f60d0 [sys.catalog]: This master's current role is: LEADER
I20260925 15:13:38.612365 19783 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea8b382304f143f4b4a3dca87c09b7e4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "01c12756ade6445f92d995ab935f60d0" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c1233ca1ad3745eca6025d08fc59d7f3" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 44383 } } peers { permanent_uuid: "ea8b382304f143f4b4a3dca87c09b7e4" member_type: VOTER last_known_addr { host: "127.12.8.61" port: 35001 } attrs { promote: false } } peers { permanent_uuid: "01c12756ade6445f92d995ab935f60d0" member_type: VOTER last_known_addr { host: "127.12.8.60" port: 41857 } attrs { promote: false } } }
I20260925 15:13:38.612632 19783 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea8b382304f143f4b4a3dca87c09b7e4 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:13:38.626958 19782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:38.619342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.1:46055 (local address 127.0.0.1:49292)
0925 15:13:38.619515 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:38.619535 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:38.619639 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:38.620844 (+ 1205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:38.620849 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:38.620873 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:13:38.621097 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:38.621107 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:38.622748 (+ 1641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:38.622754 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:38.624809 (+ 2055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:38.624822 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:38.626092 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:38.626098 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:38.626313 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:38.626374 (+ 61us) client_negotiation.cc:769] Sending connection context
0925 15:13:38.626504 (+ 130us) client_negotiation.cc:240] Negotiation successful
0925 15:13:38.626657 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260925 15:13:38.627527 19798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:38.619683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49292 (local address 127.12.8.1:46055)
0925 15:13:38.620541 (+ 858us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:38.620548 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:38.620565 (+ 17us) server_negotiation.cc:407] Connection header received
0925 15:13:38.620629 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:38.620635 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:38.620705 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:38.621227 (+ 522us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:13:38.621379 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:38.622588 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:38.625157 (+ 2569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:38.625939 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:38.626087 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:38.626301 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:38.627101 (+ 800us) server_negotiation.cc:299] Negotiation successful
0925 15:13:38.627245 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":745,"thread_start_us":644,"threads_started":1}
I20260925 15:13:38.628288 19361 tablet_service.cc:1467] Processing CreateTablet for tablet 837fd2a3b2bf441d860d128db82a8a04 (DEFAULT_TABLE table=test-table [id=893f17b4bf35496c95a919e3917c0d55]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:13:38.628718 19361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 837fd2a3b2bf441d860d128db82a8a04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:13:38.633432 19801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:38.621668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51914 (local address 127.12.8.3:40103)
0925 15:13:38.626641 (+ 4973us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:38.626647 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:38.626664 (+ 17us) server_negotiation.cc:407] Connection header received
0925 15:13:38.626724 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:38.626729 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:38.626801 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:38.626890 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:13:38.628325 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:38.629281 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:38.631300 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:38.632460 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:38.632590 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:38.632692 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:38.633012 (+ 320us) server_negotiation.cc:299] Negotiation successful
0925 15:13:38.633143 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4850,"thread_start_us":4729,"threads_started":1}
I20260925 15:13:38.633450 19800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:38.621135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.3:40103 (local address 127.0.0.1:51914)
0925 15:13:38.624683 (+ 3548us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:38.624701 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:38.624849 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:38.627767 (+ 2918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:38.627771 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:38.627794 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:13:38.628138 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:38.628148 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:38.629408 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:38.629416 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:38.631146 (+ 1730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:38.631184 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:38.632676 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:38.632682 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:38.632861 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:38.632891 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 15:13:38.632978 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 15:13:38.633105 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3409,"thread_start_us":3196,"threads_started":1}
I20260925 15:13:38.633658 19780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:38.619717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.2:35849 (local address 127.0.0.1:41398)
0925 15:13:38.619861 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:38.619876 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:38.619972 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:38.627232 (+ 7260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:38.627237 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:38.627264 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:13:38.627572 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:38.627582 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:38.628968 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:38.628972 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:38.631061 (+ 2089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:38.631080 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:38.632877 (+ 1797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:38.632884 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:38.633073 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:38.633109 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 15:13:38.633194 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 15:13:38.633331 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57,"mutex_wait_us":9}
I20260925 15:13:38.634302 19799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:38.620076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41398 (local address 127.12.8.2:35849)
0925 15:13:38.626895 (+ 6819us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:38.626901 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:38.626920 (+ 19us) server_negotiation.cc:407] Connection header received
0925 15:13:38.626988 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:38.626993 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:38.627070 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:38.627779 (+ 709us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:13:38.627917 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:38.628835 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:38.631262 (+ 2427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:38.632697 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:38.632821 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:38.632909 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:38.633842 (+ 933us) server_negotiation.cc:299] Negotiation successful
0925 15:13:38.634016 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6692,"thread_start_us":7336,"threads_started":1}
I20260925 15:13:38.634618 19646 tablet_service.cc:1467] Processing CreateTablet for tablet 837fd2a3b2bf441d860d128db82a8a04 (DEFAULT_TABLE table=test-table [id=893f17b4bf35496c95a919e3917c0d55]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:13:38.635056 19646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 837fd2a3b2bf441d860d128db82a8a04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:13:38.635035 19505 tablet_service.cc:1467] Processing CreateTablet for tablet 837fd2a3b2bf441d860d128db82a8a04 (DEFAULT_TABLE table=test-table [id=893f17b4bf35496c95a919e3917c0d55]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:13:38.635491 19505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 837fd2a3b2bf441d860d128db82a8a04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:13:38.635627 19802 tablet_bootstrap.cc:492] T 837fd2a3b2bf441d860d128db82a8a04 P 372093292e564acaae5d8d6370b2a26c: Bootstrap starting.
I20260925 15:13:38.637411 19802 tablet_bootstrap.cc:654] T 837fd2a3b2bf441d860d128db82a8a04 P 372093292e564acaae5d8d6370b2a26c: Neither blocks nor log segments found. Creating new log.
I20260925 15:13:38.640538 19804 tablet_bootstrap.cc:492] T 837fd2a3b2bf441d860d128db82a8a04 P ce72ef070a95404f8dbb29d80d7cff69: Bootstrap starting.
I20260925 15:13:38.641907 19804 tablet_bootstrap.cc:654] T 837fd2a3b2bf441d860d128db82a8a04 P ce72ef070a95404f8dbb29d80d7cff69: Neither blocks nor log segments found. Creating new log.
I20260925 15:13:38.643890 19804 tablet_bootstrap.cc:492] T 837fd2a3b2bf441d860d128db82a8a04 P ce72ef070a95404f8dbb29d80d7cff69: No bootstrap required, opened a new log
I20260925 15:13:38.644050 19804 ts_tablet_manager.cc:1397] T 837fd2a3b2bf441d860d128db82a8a04 P ce72ef070a95404f8dbb29d80d7cff69: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260925 15:13:38.644644 19804 raft_consensus.cc:348] T 837fd2a3b2bf441d860d128db82a8a04 P ce72ef070a95404f8dbb29d80d7cff69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372093292e564acaae5d8d6370b2a26c" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 46055 } } peers { permanent_uuid: "ce72ef070a95404f8dbb29d80d7cff69" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 35849 } } peers { permanent_uuid: "998c5926ac2741278b4a9c80cd3f8df5" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 40103 } }
I20260925 15:13:38.644858 19804 raft_consensus.cc:374] T 837fd2a3b2bf441d860d128db82a8a04 P ce72ef070a95404f8dbb29d80d7cff69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:13:38.644948 19804 raft_consensus.cc:729] T 837fd2a3b2bf441d860d128db82a8a04 P ce72ef070a95404f8dbb29d80d7cff69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce72ef070a95404f8dbb29d80d7cff69, State: Initialized, Role: FOLLOWER
I20260925 15:13:38.645251 19804 consensus_queue.cc:260] T 837fd2a3b2bf441d860d128db82a8a04 P ce72ef070a95404f8dbb29d80d7cff69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372093292e564acaae5d8d6370b2a26c" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 46055 } } peers { permanent_uuid: "ce72ef070a95404f8dbb29d80d7cff69" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 35849 } } peers { permanent_uuid: "998c5926ac2741278b4a9c80cd3f8df5" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 40103 } }
I20260925 15:13:38.646322 19804 ts_tablet_manager.cc:1428] T 837fd2a3b2bf441d860d128db82a8a04 P ce72ef070a95404f8dbb29d80d7cff69: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 15:13:38.647284 19808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:38.646358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:44383 (local address 127.12.8.2:36197)
0925 15:13:38.646892 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:38.647011 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:44383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":277,"threads_started":1}
I20260925 15:13:38.649166 19805 tablet_bootstrap.cc:492] T 837fd2a3b2bf441d860d128db82a8a04 P 998c5926ac2741278b4a9c80cd3f8df5: Bootstrap starting.
I20260925 15:13:38.650971 19805 tablet_bootstrap.cc:654] T 837fd2a3b2bf441d860d128db82a8a04 P 998c5926ac2741278b4a9c80cd3f8df5: Neither blocks nor log segments found. Creating new log.
I20260925 15:13:38.650929 19570 heartbeater.cc:503] Master 127.12.8.60:41857 was elected leader, sending a full tablet report...
I20260925 15:13:38.651814 19808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:38.647832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:44383 (local address 127.12.8.2:54239)
0925 15:13:38.651601 (+ 3769us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:38.651681 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:44383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3644}
I20260925 15:13:38.653450 19802 tablet_bootstrap.cc:492] T 837fd2a3b2bf441d860d128db82a8a04 P 372093292e564acaae5d8d6370b2a26c: No bootstrap required, opened a new log
I20260925 15:13:38.653613 19802 ts_tablet_manager.cc:1397] T 837fd2a3b2bf441d860d128db82a8a04 P 372093292e564acaae5d8d6370b2a26c: Time spent bootstrapping tablet: real 0.018s user 0.012s sys 0.004s
I20260925 15:13:38.653927 19805 tablet_bootstrap.cc:492] T 837fd2a3b2bf441d860d128db82a8a04 P 998c5926ac2741278b4a9c80cd3f8df5: No bootstrap required, opened a new log
I20260925 15:13:38.654064 19805 ts_tablet_manager.cc:1397] T 837fd2a3b2bf441d860d128db82a8a04 P 998c5926ac2741278b4a9c80cd3f8df5: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260925 15:13:38.654501 19802 raft_consensus.cc:348] T 837fd2a3b2bf441d860d128db82a8a04 P 372093292e564acaae5d8d6370b2a26c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372093292e564acaae5d8d6370b2a26c" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 46055 } } peers { permanent_uuid: "ce72ef070a95404f8dbb29d80d7cff69" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 35849 } } peers { permanent_uuid: "998c5926ac2741278b4a9c80cd3f8df5" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 40103 } }
I20260925 15:13:38.654771 19802 raft_consensus.cc:374] T 837fd2a3b2bf441d860d128db82a8a04 P 372093292e564acaae5d8d6370b2a26c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:13:38.654881 19802 raft_consensus.cc:729] T 837fd2a3b2bf441d860d128db82a8a04 P 372093292e564acaae5d8d6370b2a26c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 372093292e564acaae5d8d6370b2a26c, State: Initialized, Role: FOLLOWER
I20260925 15:13:38.654852 19805 raft_consensus.cc:348] T 837fd2a3b2bf441d860d128db82a8a04 P 998c5926ac2741278b4a9c80cd3f8df5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372093292e564acaae5d8d6370b2a26c" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 46055 } } peers { permanent_uuid: "ce72ef070a95404f8dbb29d80d7cff69" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 35849 } } peers { permanent_uuid: "998c5926ac2741278b4a9c80cd3f8df5" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 40103 } }
I20260925 15:13:38.655076 19805 raft_consensus.cc:374] T 837fd2a3b2bf441d860d128db82a8a04 P 998c5926ac2741278b4a9c80cd3f8df5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:13:38.655170 19805 raft_consensus.cc:729] T 837fd2a3b2bf441d860d128db82a8a04 P 998c5926ac2741278b4a9c80cd3f8df5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 998c5926ac2741278b4a9c80cd3f8df5, State: Initialized, Role: FOLLOWER
I20260925 15:13:38.655162 19802 consensus_queue.cc:260] T 837fd2a3b2bf441d860d128db82a8a04 P 372093292e564acaae5d8d6370b2a26c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372093292e564acaae5d8d6370b2a26c" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 46055 } } peers { permanent_uuid: "ce72ef070a95404f8dbb29d80d7cff69" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 35849 } } peers { permanent_uuid: "998c5926ac2741278b4a9c80cd3f8df5" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 40103 } }
I20260925 15:13:38.655400 19805 consensus_queue.cc:260] T 837fd2a3b2bf441d860d128db82a8a04 P 998c5926ac2741278b4a9c80cd3f8df5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372093292e564acaae5d8d6370b2a26c" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 46055 } } peers { permanent_uuid: "ce72ef070a95404f8dbb29d80d7cff69" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 35849 } } peers { permanent_uuid: "998c5926ac2741278b4a9c80cd3f8df5" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 40103 } }
I20260925 15:13:38.656011 19802 ts_tablet_manager.cc:1428] T 837fd2a3b2bf441d860d128db82a8a04 P 372093292e564acaae5d8d6370b2a26c: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 15:13:38.656198 19805 ts_tablet_manager.cc:1428] T 837fd2a3b2bf441d860d128db82a8a04 P 998c5926ac2741278b4a9c80cd3f8df5: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 15:13:38.659288 19426 heartbeater.cc:503] Master 127.12.8.60:41857 was elected leader, sending a full tablet report...
I20260925 15:13:38.659866 19711 heartbeater.cc:503] Master 127.12.8.60:41857 was elected leader, sending a full tablet report...
I20260925 15:13:38.663978 19813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:38.656697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:44383 (local address 127.12.8.1:38921)
0925 15:13:38.663527 (+ 6830us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:38.663680 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:44383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":6705,"thread_start_us":6558,"threads_started":1}
I20260925 15:13:38.664956 19812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:38.656676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:44383 (local address 127.12.8.3:56837)
0925 15:13:38.664579 (+ 7903us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:38.664696 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:44383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":7780,"threads_started":1}
I20260925 15:13:38.665323 19813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:38.664923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:44383 (local address 127.12.8.1:59723)
0925 15:13:38.665171 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:38.665230 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:44383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159}
I20260925 15:13:38.666375 19812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:38.666034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:44383 (local address 127.12.8.3:33621)
0925 15:13:38.666174 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:38.666285 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:44383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260925 15:13:38.906239 19810 raft_consensus.cc:482] T 837fd2a3b2bf441d860d128db82a8a04 P 372093292e564acaae5d8d6370b2a26c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:13:38.906527 19810 raft_consensus.cc:504] T 837fd2a3b2bf441d860d128db82a8a04 P 372093292e564acaae5d8d6370b2a26c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372093292e564acaae5d8d6370b2a26c" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 46055 } } peers { permanent_uuid: "ce72ef070a95404f8dbb29d80d7cff69" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 35849 } } peers { permanent_uuid: "998c5926ac2741278b4a9c80cd3f8df5" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 40103 } }
I20260925 15:13:38.908095 19810 leader_election.cc:288] T 837fd2a3b2bf441d860d128db82a8a04 P 372093292e564acaae5d8d6370b2a26c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce72ef070a95404f8dbb29d80d7cff69 (127.12.8.2:35849), 998c5926ac2741278b4a9c80cd3f8df5 (127.12.8.3:40103)
I20260925 15:13:38.915578 19799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:38.908528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.1:51163 (local address 127.12.8.2:35849)
0925 15:13:38.908734 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:38.908741 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:38.908764 (+ 23us) server_negotiation.cc:407] Connection header received
0925 15:13:38.909078 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:38.909087 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:38.909188 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:38.909299 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:13:38.909828 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:38.910942 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:38.913460 (+ 2518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:38.914657 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:38.914818 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:38.914915 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:38.915217 (+ 302us) server_negotiation.cc:299] Negotiation successful
0925 15:13:38.915332 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 15:13:38.915606 19813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:38.908442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.2:35849 (local address 127.12.8.1:51163)
0925 15:13:38.908662 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:38.908684 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:38.908881 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:38.909335 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:38.909339 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:38.909372 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:13:38.909690 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:38.909702 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:38.911057 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:38.911061 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:38.913296 (+ 2235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:38.913309 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:38.914834 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:38.914838 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:38.915001 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:38.915046 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 15:13:38.915145 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 15:13:38.915287 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62,"mutex_wait_us":96}
I20260925 15:13:38.915836 19801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:38.909004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.1:33975 (local address 127.12.8.3:40103)
0925 15:13:38.909210 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:38.909218 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:38.909631 (+ 413us) server_negotiation.cc:407] Connection header received
0925 15:13:38.909801 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:38.909807 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:38.909882 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:38.909981 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:13:38.910486 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:38.911345 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:38.913655 (+ 2310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:38.914901 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:38.915015 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:38.915080 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:38.915460 (+ 380us) server_negotiation.cc:299] Negotiation successful
0925 15:13:38.915580 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 15:13:38.916256 19524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "837fd2a3b2bf441d860d128db82a8a04" candidate_uuid: "372093292e564acaae5d8d6370b2a26c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce72ef070a95404f8dbb29d80d7cff69" is_pre_election: true
I20260925 15:13:38.916496 19814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:38.908864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.3:40103 (local address 127.12.8.1:33975)
0925 15:13:38.909521 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:38.909541 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:38.909675 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:38.910003 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:38.910006 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:38.910025 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:13:38.910326 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:38.910334 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:38.911463 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:38.911469 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:38.913517 (+ 2048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:38.913526 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:38.915104 (+ 1578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:38.915111 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:38.915287 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:38.915315 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 15:13:38.915999 (+ 684us) client_negotiation.cc:240] Negotiation successful
0925 15:13:38.916126 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"mutex_wait_us":24,"thread_start_us":369,"threads_started":1}
I20260925 15:13:38.916574 19524 raft_consensus.cc:2454] T 837fd2a3b2bf441d860d128db82a8a04 P ce72ef070a95404f8dbb29d80d7cff69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 372093292e564acaae5d8d6370b2a26c in term 0.
I20260925 15:13:38.917106 19316 leader_election.cc:302] T 837fd2a3b2bf441d860d128db82a8a04 P 372093292e564acaae5d8d6370b2a26c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 372093292e564acaae5d8d6370b2a26c, ce72ef070a95404f8dbb29d80d7cff69; no voters:
I20260925 15:13:38.917268 19666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "837fd2a3b2bf441d860d128db82a8a04" candidate_uuid: "372093292e564acaae5d8d6370b2a26c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "998c5926ac2741278b4a9c80cd3f8df5" is_pre_election: true
I20260925 15:13:38.917523 19810 raft_consensus.cc:2781] T 837fd2a3b2bf441d860d128db82a8a04 P 372093292e564acaae5d8d6370b2a26c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:13:38.917682 19810 raft_consensus.cc:482] T 837fd2a3b2bf441d860d128db82a8a04 P 372093292e564acaae5d8d6370b2a26c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:13:38.917776 19810 raft_consensus.cc:3037] T 837fd2a3b2bf441d860d128db82a8a04 P 372093292e564acaae5d8d6370b2a26c [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:13:38.917702 19666 raft_consensus.cc:2454] T 837fd2a3b2bf441d860d128db82a8a04 P 998c5926ac2741278b4a9c80cd3f8df5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 372093292e564acaae5d8d6370b2a26c in term 0.
I20260925 15:13:38.919996 19810 raft_consensus.cc:504] T 837fd2a3b2bf441d860d128db82a8a04 P 372093292e564acaae5d8d6370b2a26c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372093292e564acaae5d8d6370b2a26c" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 46055 } } peers { permanent_uuid: "ce72ef070a95404f8dbb29d80d7cff69" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 35849 } } peers { permanent_uuid: "998c5926ac2741278b4a9c80cd3f8df5" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 40103 } }
I20260925 15:13:38.920549 19810 leader_election.cc:288] T 837fd2a3b2bf441d860d128db82a8a04 P 372093292e564acaae5d8d6370b2a26c [CANDIDATE]: Term 1 election: Requested vote from peers ce72ef070a95404f8dbb29d80d7cff69 (127.12.8.2:35849), 998c5926ac2741278b4a9c80cd3f8df5 (127.12.8.3:40103)
I20260925 15:13:38.920991 19524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "837fd2a3b2bf441d860d128db82a8a04" candidate_uuid: "372093292e564acaae5d8d6370b2a26c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce72ef070a95404f8dbb29d80d7cff69"
I20260925 15:13:38.921057 19666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "837fd2a3b2bf441d860d128db82a8a04" candidate_uuid: "372093292e564acaae5d8d6370b2a26c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "998c5926ac2741278b4a9c80cd3f8df5"
I20260925 15:13:38.921191 19524 raft_consensus.cc:3037] T 837fd2a3b2bf441d860d128db82a8a04 P ce72ef070a95404f8dbb29d80d7cff69 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:13:38.921236 19666 raft_consensus.cc:3037] T 837fd2a3b2bf441d860d128db82a8a04 P 998c5926ac2741278b4a9c80cd3f8df5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:13:38.923722 19524 raft_consensus.cc:2454] T 837fd2a3b2bf441d860d128db82a8a04 P ce72ef070a95404f8dbb29d80d7cff69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 372093292e564acaae5d8d6370b2a26c in term 1.
I20260925 15:13:38.923765 19666 raft_consensus.cc:2454] T 837fd2a3b2bf441d860d128db82a8a04 P 998c5926ac2741278b4a9c80cd3f8df5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 372093292e564acaae5d8d6370b2a26c in term 1.
I20260925 15:13:38.924108 19316 leader_election.cc:302] T 837fd2a3b2bf441d860d128db82a8a04 P 372093292e564acaae5d8d6370b2a26c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 372093292e564acaae5d8d6370b2a26c, ce72ef070a95404f8dbb29d80d7cff69; no voters:
I20260925 15:13:38.924458 19810 raft_consensus.cc:2781] T 837fd2a3b2bf441d860d128db82a8a04 P 372093292e564acaae5d8d6370b2a26c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:13:38.925038 19810 raft_consensus.cc:686] T 837fd2a3b2bf441d860d128db82a8a04 P 372093292e564acaae5d8d6370b2a26c [term 1 LEADER]: Becoming Leader. State: Replica: 372093292e564acaae5d8d6370b2a26c, State: Running, Role: LEADER
I20260925 15:13:38.925407 19810 consensus_queue.cc:237] T 837fd2a3b2bf441d860d128db82a8a04 P 372093292e564acaae5d8d6370b2a26c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372093292e564acaae5d8d6370b2a26c" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 46055 } } peers { permanent_uuid: "ce72ef070a95404f8dbb29d80d7cff69" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 35849 } } peers { permanent_uuid: "998c5926ac2741278b4a9c80cd3f8df5" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 40103 } }
I20260925 15:13:38.926301 19814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:38.925843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:44383 (local address 127.12.8.1:37551)
0925 15:13:38.925984 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:38.926112 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:44383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260925 15:13:38.929229 19248 catalog_manager.cc:5495] T 837fd2a3b2bf441d860d128db82a8a04 P 372093292e564acaae5d8d6370b2a26c reported cstate change: term changed from 0 to 1, leader changed from <none> to 372093292e564acaae5d8d6370b2a26c (127.12.8.1). New cstate: current_term: 1 leader_uuid: "372093292e564acaae5d8d6370b2a26c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372093292e564acaae5d8d6370b2a26c" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 46055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce72ef070a95404f8dbb29d80d7cff69" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 35849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "998c5926ac2741278b4a9c80cd3f8df5" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 40103 } health_report { overall_health: UNKNOWN } } }
I20260925 15:13:38.998302 19798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:38.993455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49298 (local address 127.12.8.1:46055)
0925 15:13:38.993658 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:38.993662 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:38.993680 (+ 18us) server_negotiation.cc:407] Connection header received
0925 15:13:38.993811 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:38.993815 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:38.993889 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:38.994031 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:13:38.994379 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:38.995073 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:38.996371 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:38.996834 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:38.996944 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:38.997280 (+ 336us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:13:38.997783 (+ 503us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:13:38.997894 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:38.997985 (+ 91us) server_negotiation.cc:299] Negotiation successful
0925 15:13:38.998046 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 15:13:39.002154 19524 raft_consensus.cc:1261] T 837fd2a3b2bf441d860d128db82a8a04 P ce72ef070a95404f8dbb29d80d7cff69 [term 1 FOLLOWER]: Refusing update from remote peer 372093292e564acaae5d8d6370b2a26c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:13:39.002643 19666 raft_consensus.cc:1261] T 837fd2a3b2bf441d860d128db82a8a04 P 998c5926ac2741278b4a9c80cd3f8df5 [term 1 FOLLOWER]: Refusing update from remote peer 372093292e564acaae5d8d6370b2a26c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:13:39.002923 19815 consensus_queue.cc:1035] T 837fd2a3b2bf441d860d128db82a8a04 P 372093292e564acaae5d8d6370b2a26c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce72ef070a95404f8dbb29d80d7cff69" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 35849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:13:39.003332 19810 consensus_queue.cc:1035] T 837fd2a3b2bf441d860d128db82a8a04 P 372093292e564acaae5d8d6370b2a26c [LEADER]: Connected to new peer: Peer: permanent_uuid: "998c5926ac2741278b4a9c80cd3f8df5" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 40103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:13:39.005745 19808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:39.005215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:44383 (local address 127.12.8.2:33675)
0925 15:13:39.005436 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:39.005554 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:44383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260925 15:13:39.006274 19812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:39.005260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:44383 (local address 127.12.8.3:42545)
0925 15:13:39.005527 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:39.005662 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:44383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
I20260925 15:13:39.016711 19812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:39.016238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:44383 (local address 127.12.8.3:34333)
0925 15:13:39.016462 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:39.016554 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:44383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
I20260925 15:13:39.016710 19808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:39.016238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:44383 (local address 127.12.8.2:56337)
0925 15:13:39.016469 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:39.016554 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:44383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260925 15:13:39.017204 19814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:39.016720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:44383 (local address 127.12.8.1:34147)
0925 15:13:39.016953 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:39.017050 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:44383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
I20260925 15:13:39.028496 19248 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51160:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:13:39.028774 19248 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51160:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:13:39.033955 19248 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 01c12756ade6445f92d995ab935f60d0: Sending DeleteTablet for 3 replicas of tablet 837fd2a3b2bf441d860d128db82a8a04
I20260925 15:13:39.034847 19361 tablet_service.cc:1514] Processing DeleteTablet for tablet 837fd2a3b2bf441d860d128db82a8a04 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:13:39 UTC) from {username='oryx', principal='oryx/127.12.8.60@KRBTEST.COM'} at 127.0.0.1:49292
I20260925 15:13:39.034953 19505 tablet_service.cc:1514] Processing DeleteTablet for tablet 837fd2a3b2bf441d860d128db82a8a04 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:13:39 UTC) from {username='oryx', principal='oryx/127.12.8.60@KRBTEST.COM'} at 127.0.0.1:41398
I20260925 15:13:39.035194 19646 tablet_service.cc:1514] Processing DeleteTablet for tablet 837fd2a3b2bf441d860d128db82a8a04 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:13:39 UTC) from {username='oryx', principal='oryx/127.12.8.60@KRBTEST.COM'} at 127.0.0.1:51914
I20260925 15:13:39.038266 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 19220
I20260925 15:13:39.042699 19830 tablet_replica.cc:330] T 837fd2a3b2bf441d860d128db82a8a04 P 998c5926ac2741278b4a9c80cd3f8df5: stopping tablet replica
I20260925 15:13:39.042968 19830 raft_consensus.cc:2229] T 837fd2a3b2bf441d860d128db82a8a04 P 998c5926ac2741278b4a9c80cd3f8df5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:13:39.043311 19830 raft_consensus.cc:2258] T 837fd2a3b2bf441d860d128db82a8a04 P 998c5926ac2741278b4a9c80cd3f8df5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:13:39.047019 19830 ts_tablet_manager.cc:1905] T 837fd2a3b2bf441d860d128db82a8a04 P 998c5926ac2741278b4a9c80cd3f8df5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:13:39.050022 19829 tablet_replica.cc:330] T 837fd2a3b2bf441d860d128db82a8a04 P ce72ef070a95404f8dbb29d80d7cff69: stopping tablet replica
I20260925 15:13:39.050246 19829 raft_consensus.cc:2229] T 837fd2a3b2bf441d860d128db82a8a04 P ce72ef070a95404f8dbb29d80d7cff69 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:13:39.050536 19829 raft_consensus.cc:2258] T 837fd2a3b2bf441d860d128db82a8a04 P ce72ef070a95404f8dbb29d80d7cff69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:13:39.053916 19780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:39.053473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:44383 (local address 127.0.0.1:32996)
0925 15:13:39.053681 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:39.053762 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:44383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260925 15:13:39.056386 19780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:39.055994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:44383 (local address 127.0.0.1:33008)
0925 15:13:39.056174 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:39.056249 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:44383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260925 15:13:39.058257 19828 tablet_replica.cc:330] T 837fd2a3b2bf441d860d128db82a8a04 P 372093292e564acaae5d8d6370b2a26c: stopping tablet replica
I20260925 15:13:39.058452 19828 raft_consensus.cc:2229] T 837fd2a3b2bf441d860d128db82a8a04 P 372093292e564acaae5d8d6370b2a26c [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:13:39.058525 19829 ts_tablet_manager.cc:1905] T 837fd2a3b2bf441d860d128db82a8a04 P ce72ef070a95404f8dbb29d80d7cff69: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:13:39.059092 19828 raft_consensus.cc:2258] T 837fd2a3b2bf441d860d128db82a8a04 P 372093292e564acaae5d8d6370b2a26c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:13:39.063012 19828 ts_tablet_manager.cc:1905] T 837fd2a3b2bf441d860d128db82a8a04 P 372093292e564acaae5d8d6370b2a26c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:13:39.065367 19829 ts_tablet_manager.cc:1918] T 837fd2a3b2bf441d860d128db82a8a04 P ce72ef070a95404f8dbb29d80d7cff69: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:13:39.065518 19829 log.cc:1196] T 837fd2a3b2bf441d860d128db82a8a04 P ce72ef070a95404f8dbb29d80d7cff69: Deleting WAL directory at /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/ts-1/wal/wals/837fd2a3b2bf441d860d128db82a8a04
I20260925 15:13:39.065956 19829 ts_tablet_manager.cc:1939] T 837fd2a3b2bf441d860d128db82a8a04 P ce72ef070a95404f8dbb29d80d7cff69: Deleting consensus metadata
I20260925 15:13:39.066007 19830 ts_tablet_manager.cc:1918] T 837fd2a3b2bf441d860d128db82a8a04 P 998c5926ac2741278b4a9c80cd3f8df5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:13:39.066156 19830 log.cc:1196] T 837fd2a3b2bf441d860d128db82a8a04 P 998c5926ac2741278b4a9c80cd3f8df5: Deleting WAL directory at /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/ts-2/wal/wals/837fd2a3b2bf441d860d128db82a8a04
I20260925 15:13:39.066624 19830 ts_tablet_manager.cc:1939] T 837fd2a3b2bf441d860d128db82a8a04 P 998c5926ac2741278b4a9c80cd3f8df5: Deleting consensus metadata
I20260925 15:13:39.067056 19234 catalog_manager.cc:4841] TS ce72ef070a95404f8dbb29d80d7cff69 (127.12.8.2:35849): tablet 837fd2a3b2bf441d860d128db82a8a04 (table test-table [id=893f17b4bf35496c95a919e3917c0d55]) successfully deleted
I20260925 15:13:39.067613 19234 catalog_manager.cc:4841] TS 998c5926ac2741278b4a9c80cd3f8df5 (127.12.8.3:40103): tablet 837fd2a3b2bf441d860d128db82a8a04 (table test-table [id=893f17b4bf35496c95a919e3917c0d55]) successfully deleted
I20260925 15:13:39.069371 19828 ts_tablet_manager.cc:1918] T 837fd2a3b2bf441d860d128db82a8a04 P 372093292e564acaae5d8d6370b2a26c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:13:39.069504 19828 log.cc:1196] T 837fd2a3b2bf441d860d128db82a8a04 P 372093292e564acaae5d8d6370b2a26c: Deleting WAL directory at /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349162370577-12320-0/minicluster-data/ts-0/wal/wals/837fd2a3b2bf441d860d128db82a8a04
I20260925 15:13:39.069967 19828 ts_tablet_manager.cc:1939] T 837fd2a3b2bf441d860d128db82a8a04 P 372093292e564acaae5d8d6370b2a26c: Deleting consensus metadata
I20260925 15:13:39.071030 19232 catalog_manager.cc:4841] TS 372093292e564acaae5d8d6370b2a26c (127.12.8.1:46055): tablet 837fd2a3b2bf441d860d128db82a8a04 (table test-table [id=893f17b4bf35496c95a919e3917c0d55]) successfully deleted
I20260925 15:13:39.076896 19798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:39.039262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49306 (local address 127.12.8.1:46055)
0925 15:13:39.039427 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:39.039433 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:39.051176 (+ 11743us) server_negotiation.cc:407] Connection header received
0925 15:13:39.051365 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:39.051370 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:39.051479 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:39.051586 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:39.054507 (+ 2921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:39.055674 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:39.057457 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:39.058205 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:39.058324 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:39.070731 (+ 12407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:39.070746 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:39.070767 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:39.070806 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:39.074005 (+ 3199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:39.074711 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:39.074721 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:39.074725 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:39.074892 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:39.075334 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:39.075340 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:39.075343 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:39.075786 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:39.076128 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:39.076282 (+ 154us) server_negotiation.cc:299] Negotiation successful
0925 15:13:39.076506 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 15:13:39.077586 19421 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:13:39.179078 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 19220
I20260925 15:13:39.197835 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 19437
I20260925 15:13:39.210737 19799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:39.198640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41408 (local address 127.12.8.2:35849)
0925 15:13:39.198799 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:39.198803 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:39.198884 (+ 81us) server_negotiation.cc:407] Connection header received
0925 15:13:39.199058 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:39.199062 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:39.199130 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:39.199212 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:39.200439 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:39.201146 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:39.202636 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:39.203009 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:39.203101 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:39.205418 (+ 2317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:39.205433 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:39.205449 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:39.205491 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:39.208324 (+ 2833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:39.208899 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:39.208907 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:39.208909 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:39.209040 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:39.209387 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:39.209390 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:39.209392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:39.209729 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:39.209929 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:39.210161 (+ 232us) server_negotiation.cc:299] Negotiation successful
0925 15:13:39.210463 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 15:13:39.211357 19565 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:13:39.303834 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 19437
I20260925 15:13:39.321766 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 19583
I20260925 15:13:39.335608 19801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:39.322576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51930 (local address 127.12.8.3:40103)
0925 15:13:39.322751 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:39.322757 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:39.322807 (+ 50us) server_negotiation.cc:407] Connection header received
0925 15:13:39.323044 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:39.323048 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:39.323111 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:39.323215 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:39.324721 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:39.325587 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:39.326984 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:39.327366 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:39.327506 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:39.329991 (+ 2485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:39.330005 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:39.330022 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:39.330066 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:39.332863 (+ 2797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:39.333520 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:39.333526 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:39.333528 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:39.333652 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:39.334045 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:39.334054 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:39.334056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:39.334468 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:39.334790 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:39.335073 (+ 283us) server_negotiation.cc:299] Negotiation successful
0925 15:13:39.335285 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 15:13:39.336295 19706 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:13:39.452646 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 19583
I20260925 15:13:39.456321 19780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:39.455922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:44383 (local address 127.0.0.1:33014)
0925 15:13:39.456096 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:39.456177 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:44383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260925 15:13:39.457273 19780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:39.456982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:44383 (local address 127.0.0.1:33022)
0925 15:13:39.457103 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:39.457175 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:44383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260925 15:13:39.473858 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 18922
I20260925 15:13:39.474953 19088 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:13:39.588770 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 18922
I20260925 15:13:39.608734 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 19030
I20260925 15:13:39.609647 19278 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:13:39.727835 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 19030
2026-09-25T15:13:39Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9500 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:13:39 dist-test-slave-p468 krb5kdc[19839](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:13:39 dist-test-slave-p468 krb5kdc[19839](info): set up 2 sockets
Sep 25 15:13:39 dist-test-slave-p468 krb5kdc[19839](info): commencing operation
krb5kdc: starting...
W20260925 15:13:40.819177 12320 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.001s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/krb5kdc/test-user.keytab.
Sep 25 15:13:40 dist-test-slave-p468 krb5kdc[19839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:13:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:13:40Z Disabled control of system clock
WARNING: no policy specified for oryx/127.12.8.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.8.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.8.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/krb5kdc/oryx_127.12.8.62.keytab.
Entry for principal oryx/127.12.8.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/krb5kdc/oryx_127.12.8.62.keytab.
I20260925 15:13:41.029742 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.62:32861
--webserver_interface=127.12.8.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/krb5kdc/oryx_127.12.8.62.keytab
--principal=oryx/127.12.8.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:35675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.62:32861
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:13:41.342657 19852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:13:41.342945 19852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:13:41.343110 19852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:13:41.350281 19852 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:13:41.350381 19852 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 15:13:41.350477 19852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:13:41.350565 19852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:13:41.350677 19852 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:13:41.350776 19852 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:13:41.360527 19852 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:35675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.8.62:32861
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/krb5kdc/oryx_127.12.8.62.keytab
--principal=oryx/127.12.8.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.62:32861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.8.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:13:41.361832 19852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:13:41.363224 19852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:13:41.374979 19860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:41.375094 19857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:41.375154 19858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:13:41.376353 19852 server_base.cc:1034] running on GCE node
I20260925 15:13:41.377063 19852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:13:41.378443 19852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:13:41.379644 19852 hybrid_clock.cc:648] HybridClock initialized: now 1790349221379578 us; error 42 us; skew 500 ppm
I20260925 15:13:41.379966 19852 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:13:41 dist-test-slave-p468 krb5kdc[19839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349221, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.8.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:13:41.383011 19852 init.cc:377] Logged in from keytab as oryx/127.12.8.62@KRBTEST.COM (short username oryx)
I20260925 15:13:41.384440 19852 webserver.cc:460] Webserver started at http://127.12.8.62:37237/ using document root <none> and password file <none>
I20260925 15:13:41.385054 19852 fs_manager.cc:359] Metadata directory not provided
I20260925 15:13:41.385135 19852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:13:41.385391 19852 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:13:41.388258 19852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/minicluster-data/master-0/data/instance:
uuid: "3312e9eb760f4482923279ed63ae02a8"
format_stamp: "Formatted at 2026-09-25 15:13:41 on dist-test-slave-p468"
server_key: "17740d4406bc6074d7b7e70bdf31fcbf"
server_key_iv: "cbb83c19a3050928f502d7b0586a6b9a"
server_key_version: "encryptionkey@0"
I20260925 15:13:41.389026 19852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/minicluster-data/master-0/wal/instance:
uuid: "3312e9eb760f4482923279ed63ae02a8"
format_stamp: "Formatted at 2026-09-25 15:13:41 on dist-test-slave-p468"
server_key: "17740d4406bc6074d7b7e70bdf31fcbf"
server_key_iv: "cbb83c19a3050928f502d7b0586a6b9a"
server_key_version: "encryptionkey@0"
I20260925 15:13:41.394007 19852 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.006s
I20260925 15:13:41.397384 19867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:41.398790 19852 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:13:41.398972 19852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/minicluster-data/master-0/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/minicluster-data/master-0/wal
uuid: "3312e9eb760f4482923279ed63ae02a8"
format_stamp: "Formatted at 2026-09-25 15:13:41 on dist-test-slave-p468"
server_key: "17740d4406bc6074d7b7e70bdf31fcbf"
server_key_iv: "cbb83c19a3050928f502d7b0586a6b9a"
server_key_version: "encryptionkey@0"
I20260925 15:13:41.399189 19852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:13:41.432137 19852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:13:41.436061 19852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:13:41.436327 19852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:13:41.458343 19852 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.62:32861
I20260925 15:13:41.458496 19928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.62:32861 every 8 connection(s)
I20260925 15:13:41.459738 19852 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
I20260925 15:13:41.463547 19929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:13:41.469267 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 19852
I20260925 15:13:41.469491 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/minicluster-data/master-0/wal/instance
I20260925 15:13:41.470144 12320 external_mini_cluster.cc:1428] Setting key 3d5e276e2c964a5efd9dcd21f51bd695
I20260925 15:13:41.476289 19929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3312e9eb760f4482923279ed63ae02a8: Bootstrap starting.
I20260925 15:13:41.480563 19929 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3312e9eb760f4482923279ed63ae02a8: Neither blocks nor log segments found. Creating new log.
I20260925 15:13:41.481918 19929 log.cc:824] T 00000000000000000000000000000000 P 3312e9eb760f4482923279ed63ae02a8: Log is configured to *not* fsync() on all Append() calls
I20260925 15:13:41.485783 19929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3312e9eb760f4482923279ed63ae02a8: No bootstrap required, opened a new log
Sep 25 15:13:41 dist-test-slave-p468 krb5kdc[19839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM
I20260925 15:13:41.497486 19929 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3312e9eb760f4482923279ed63ae02a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3312e9eb760f4482923279ed63ae02a8" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 32861 } }
I20260925 15:13:41.497998 19929 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3312e9eb760f4482923279ed63ae02a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:13:41.498124 19929 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3312e9eb760f4482923279ed63ae02a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3312e9eb760f4482923279ed63ae02a8, State: Initialized, Role: FOLLOWER
I20260925 15:13:41.498579 19932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:41.472492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58184 (local address 127.12.8.62:32861)
0925 15:13:41.473639 (+ 1147us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:41.473656 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:41.474053 (+ 397us) server_negotiation.cc:407] Connection header received
0925 15:13:41.476805 (+ 2752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:41.476835 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:41.477404 (+ 569us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:41.477872 (+ 468us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:41.479228 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:41.481443 (+ 2215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:41.483463 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:41.484588 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:41.484803 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:41.489118 (+ 4315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:41.489131 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:41.489153 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:41.489223 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:41.493526 (+ 4303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:41.494253 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:41.494263 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:41.494273 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:41.494497 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:41.494901 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:41.494906 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:41.494910 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:41.495738 (+ 828us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:41.496058 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:41.496499 (+ 441us) server_negotiation.cc:299] Negotiation successful
0925 15:13:41.496962 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":851,"thread_start_us":531,"threads_started":1}
I20260925 15:13:41.498914 19929 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3312e9eb760f4482923279ed63ae02a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3312e9eb760f4482923279ed63ae02a8" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 32861 } }
I20260925 15:13:41.499332 19929 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3312e9eb760f4482923279ed63ae02a8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:13:41.499557 19929 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3312e9eb760f4482923279ed63ae02a8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:13:41.499809 19929 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3312e9eb760f4482923279ed63ae02a8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:13:41.503082 19929 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3312e9eb760f4482923279ed63ae02a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3312e9eb760f4482923279ed63ae02a8" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 32861 } }
I20260925 15:13:41.503695 19929 leader_election.cc:302] T 00000000000000000000000000000000 P 3312e9eb760f4482923279ed63ae02a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3312e9eb760f4482923279ed63ae02a8; no voters:
I20260925 15:13:41.504302 19929 leader_election.cc:288] T 00000000000000000000000000000000 P 3312e9eb760f4482923279ed63ae02a8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:13:41.504484 19934 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3312e9eb760f4482923279ed63ae02a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:13:41.505362 19934 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3312e9eb760f4482923279ed63ae02a8 [term 1 LEADER]: Becoming Leader. State: Replica: 3312e9eb760f4482923279ed63ae02a8, State: Running, Role: LEADER
I20260925 15:13:41.505914 19934 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3312e9eb760f4482923279ed63ae02a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3312e9eb760f4482923279ed63ae02a8" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 32861 } }
I20260925 15:13:41.506563 19929 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3312e9eb760f4482923279ed63ae02a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:13:41.508757 19935 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3312e9eb760f4482923279ed63ae02a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3312e9eb760f4482923279ed63ae02a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3312e9eb760f4482923279ed63ae02a8" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 32861 } } }
I20260925 15:13:41.509182 19935 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3312e9eb760f4482923279ed63ae02a8 [sys.catalog]: This master's current role is: LEADER
I20260925 15:13:41.509254 19936 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3312e9eb760f4482923279ed63ae02a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3312e9eb760f4482923279ed63ae02a8. Latest consensus state: current_term: 1 leader_uuid: "3312e9eb760f4482923279ed63ae02a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3312e9eb760f4482923279ed63ae02a8" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 32861 } } }
I20260925 15:13:41.509522 19936 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3312e9eb760f4482923279ed63ae02a8 [sys.catalog]: This master's current role is: LEADER
I20260925 15:13:41.511341 19939 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:13:41.519418 19939 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:13:41.530078 19939 catalog_manager.cc:1347] Generated new cluster ID: 98e0bd44455f4dd7ae4580500a0a88ed
I20260925 15:13:41.530224 19939 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:13:41.585978 19939 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:13:41.588049 19939 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:13:41.594815 19939 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3312e9eb760f4482923279ed63ae02a8: Generated new TSK 0
I20260925 15:13:41.595876 19939 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.12.8.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.8.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.8.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/krb5kdc/oryx_127.12.8.1.keytab.
Entry for principal oryx/127.12.8.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/krb5kdc/oryx_127.12.8.1.keytab.
I20260925 15:13:41.661549 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.1:0
--local_ip_for_outbound_sockets=127.12.8.1
--webserver_interface=127.12.8.1
--webserver_port=0
--tserver_master_addrs=127.12.8.62:32861
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/krb5kdc/oryx_127.12.8.1.keytab
--principal=oryx/127.12.8.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:35675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:13:41.987360 19955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:13:41.987673 19955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:13:41.987843 19955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:13:41.995025 19955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:13:41.995178 19955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:13:41.995321 19955 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 15:13:41.995463 19955 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.1
I20260925 15:13:42.005384 19955 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:35675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/krb5kdc/oryx_127.12.8.1.keytab
--principal=oryx/127.12.8.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.8.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.8.62:32861
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.1
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:13:42.006759 19955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:13:42.008153 19955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:13:42.021245 19960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:42.021277 19961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:42.021216 19963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:13:42.022293 19955 server_base.cc:1034] running on GCE node
I20260925 15:13:42.022889 19955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:13:42.023769 19955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:13:42.024989 19955 hybrid_clock.cc:648] HybridClock initialized: now 1790349222024927 us; error 31 us; skew 500 ppm
I20260925 15:13:42.025288 19955 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:13:42 dist-test-slave-p468 krb5kdc[19839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349222, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.8.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:13:42.028544 19955 init.cc:377] Logged in from keytab as oryx/127.12.8.1@KRBTEST.COM (short username oryx)
I20260925 15:13:42.029990 19955 webserver.cc:460] Webserver started at http://127.12.8.1:43419/ using document root <none> and password file <none>
I20260925 15:13:42.030740 19955 fs_manager.cc:359] Metadata directory not provided
I20260925 15:13:42.030874 19955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:13:42.031195 19955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:13:42.034016 19955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/minicluster-data/ts-0/data/instance:
uuid: "c2e6c5753e2546d9bc00393d25672551"
format_stamp: "Formatted at 2026-09-25 15:13:42 on dist-test-slave-p468"
server_key: "832122be8636e7f394be082813f2b294"
server_key_iv: "c08f5a0691a4ec6b807c481a274704af"
server_key_version: "encryptionkey@0"
I20260925 15:13:42.034879 19955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/minicluster-data/ts-0/wal/instance:
uuid: "c2e6c5753e2546d9bc00393d25672551"
format_stamp: "Formatted at 2026-09-25 15:13:42 on dist-test-slave-p468"
server_key: "832122be8636e7f394be082813f2b294"
server_key_iv: "c08f5a0691a4ec6b807c481a274704af"
server_key_version: "encryptionkey@0"
I20260925 15:13:42.039718 19955 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 15:13:42.042984 19970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:42.044157 19955 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s
I20260925 15:13:42.044322 19955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/minicluster-data/ts-0/wal
uuid: "c2e6c5753e2546d9bc00393d25672551"
format_stamp: "Formatted at 2026-09-25 15:13:42 on dist-test-slave-p468"
server_key: "832122be8636e7f394be082813f2b294"
server_key_iv: "c08f5a0691a4ec6b807c481a274704af"
server_key_version: "encryptionkey@0"
I20260925 15:13:42.044552 19955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:13:42.064836 19955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:13:42.068570 19955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:13:42.068843 19955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:13:42.071487 19955 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:13:42.071604 19955 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:42.071808 19955 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:13:42.071894 19955 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 15:13:42 dist-test-slave-p468 krb5kdc[19839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349222, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.8.1@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM
I20260925 15:13:42.093334 19978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:42.071817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.1:60469 (local address 127.12.8.62:32861)
0925 15:13:42.072296 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:42.072302 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:42.073527 (+ 1225us) server_negotiation.cc:407] Connection header received
0925 15:13:42.075841 (+ 2314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:42.075847 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:42.075946 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:42.076065 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:42.078220 (+ 2155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:42.079513 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:42.081964 (+ 2451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:42.082592 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:42.082734 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:42.087555 (+ 4821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:42.087560 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:42.087563 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:42.087593 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:42.090049 (+ 2456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:42.090967 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:42.090979 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:42.090982 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:42.091101 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:42.091554 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:42.091559 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:42.091563 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:42.091956 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:42.092111 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:42.092755 (+ 644us) server_negotiation.cc:299] Negotiation successful
0925 15:13:42.092978 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":249,"threads_started":1}
I20260925 15:13:42.095634 19980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:42.072463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:32861 (local address 127.12.8.1:60469)
0925 15:13:42.073341 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:42.073398 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:42.075578 (+ 2180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:42.076326 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:42.076340 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:42.077133 (+ 793us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:13:42.077933 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:42.077955 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:42.079697 (+ 1742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:42.079705 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:42.081816 (+ 2111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:42.081828 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:42.082848 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:42.082864 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:42.083083 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:42.083915 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:13:42.083936 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:13:42.087328 (+ 3392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:13:42.090183 (+ 2855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:13:42.090213 (+ 30us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:13:42.090228 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:13:42.090790 (+ 562us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:13:42.091210 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:13:42.091217 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:13:42.091220 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:13:42.091401 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:13:42.092089 (+ 688us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:13:42.092097 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:13:42.092421 (+ 324us) client_negotiation.cc:769] Sending connection context
0925 15:13:42.092666 (+ 245us) client_negotiation.cc:240] Negotiation successful
0925 15:13:42.093137 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"thread_start_us":378,"threads_started":1}
I20260925 15:13:42.116950 19978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:42.101499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.1:58119 (local address 127.12.8.62:32861)
0925 15:13:42.101706 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:42.101714 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:42.102345 (+ 631us) server_negotiation.cc:407] Connection header received
0925 15:13:42.102533 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:42.102536 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:42.102606 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:42.102704 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:42.104068 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:42.104739 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:42.107250 (+ 2511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:42.107828 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:42.107982 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:42.110826 (+ 2844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:42.110835 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:42.110841 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:42.110878 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:42.113768 (+ 2890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:42.114828 (+ 1060us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:42.114836 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:42.114840 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:42.115014 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:42.115498 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:42.115505 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:42.115509 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:42.115837 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:42.116004 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:42.116390 (+ 386us) server_negotiation.cc:299] Negotiation successful
0925 15:13:42.116601 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 15:13:42.117041 20035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:42.101464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:32861 (local address 127.12.8.1:58119)
0925 15:13:42.102224 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:42.102254 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:42.102417 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:42.102750 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:42.102758 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:42.103466 (+ 708us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:13:42.103921 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:42.103936 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:42.104872 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:42.104881 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:42.107025 (+ 2144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:42.107048 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:42.108058 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:42.108067 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:42.108362 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:42.109299 (+ 937us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:13:42.109331 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:13:42.110617 (+ 1286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:13:42.113918 (+ 3301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:13:42.113921 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:13:42.113924 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:13:42.114609 (+ 685us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:13:42.115159 (+ 550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:13:42.115165 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:13:42.115168 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:13:42.115307 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:13:42.115966 (+ 659us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:13:42.115971 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:13:42.116145 (+ 174us) client_negotiation.cc:769] Sending connection context
0925 15:13:42.116282 (+ 137us) client_negotiation.cc:240] Negotiation successful
0925 15:13:42.116592 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":395,"threads_started":1}
I20260925 15:13:42.130682 19955 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.1:39179
I20260925 15:13:42.130700 20090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.1:39179 every 8 connection(s)
I20260925 15:13:42.131846 19955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/minicluster-data/ts-0/data/info.pb
I20260925 15:13:42.133733 20091 heartbeater.cc:347] Connected to a master server at 127.12.8.62:32861
I20260925 15:13:42.133921 20091 heartbeater.cc:464] Registering TS with master...
I20260925 15:13:42.134565 20091 heartbeater.cc:511] Master 127.12.8.62:32861 requested a full tablet report, sending...
I20260925 15:13:42.135547 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 19955
I20260925 15:13:42.135779 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/minicluster-data/ts-0/wal/instance
I20260925 15:13:42.136442 12320 external_mini_cluster.cc:1428] Setting key a90b0894ac1ccdd9be94220239d898be
I20260925 15:13:42.136643 19884 ts_manager.cc:194] Registered new tserver with Master: c2e6c5753e2546d9bc00393d25672551 (127.12.8.1:39179)
I20260925 15:13:42.138491 19884 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.8.1@KRBTEST.COM'} at 127.12.8.1:60469
WARNING: no policy specified for oryx/127.12.8.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.8.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.8.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/krb5kdc/oryx_127.12.8.2.keytab.
Entry for principal oryx/127.12.8.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/krb5kdc/oryx_127.12.8.2.keytab.
I20260925 15:13:42.182391 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.2:0
--local_ip_for_outbound_sockets=127.12.8.2
--webserver_interface=127.12.8.2
--webserver_port=0
--tserver_master_addrs=127.12.8.62:32861
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/krb5kdc/oryx_127.12.8.2.keytab
--principal=oryx/127.12.8.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:35675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:13:42.503909 20095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:13:42.504209 20095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:13:42.504385 20095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:13:42.511713 20095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:13:42.511855 20095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:13:42.512040 20095 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 15:13:42.512182 20095 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.2
I20260925 15:13:42.521786 20095 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:35675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/krb5kdc/oryx_127.12.8.2.keytab
--principal=oryx/127.12.8.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.8.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.8.62:32861
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.2
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:13:42.523089 20095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:13:42.524420 20095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:13:42.537261 20101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:42.537350 20100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:42.537374 20103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:13:42.538245 20095 server_base.cc:1034] running on GCE node
I20260925 15:13:42.538767 20095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:13:42.539577 20095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:13:42.540781 20095 hybrid_clock.cc:648] HybridClock initialized: now 1790349222540722 us; error 32 us; skew 500 ppm
I20260925 15:13:42.541085 20095 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:13:42 dist-test-slave-p468 krb5kdc[19839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349222, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.8.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:13:42.544173 20095 init.cc:377] Logged in from keytab as oryx/127.12.8.2@KRBTEST.COM (short username oryx)
I20260925 15:13:42.545513 20095 webserver.cc:460] Webserver started at http://127.12.8.2:43403/ using document root <none> and password file <none>
I20260925 15:13:42.546269 20095 fs_manager.cc:359] Metadata directory not provided
I20260925 15:13:42.546397 20095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:13:42.546710 20095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:13:42.549556 20095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/minicluster-data/ts-1/data/instance:
uuid: "bf7cab7961534d01af401cdd0fc4c5ff"
format_stamp: "Formatted at 2026-09-25 15:13:42 on dist-test-slave-p468"
server_key: "0af7098b75bd7b103f9af24efbfad020"
server_key_iv: "7bf7062c6eaaf9acaf18efb753b70e4e"
server_key_version: "encryptionkey@0"
I20260925 15:13:42.550403 20095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/minicluster-data/ts-1/wal/instance:
uuid: "bf7cab7961534d01af401cdd0fc4c5ff"
format_stamp: "Formatted at 2026-09-25 15:13:42 on dist-test-slave-p468"
server_key: "0af7098b75bd7b103f9af24efbfad020"
server_key_iv: "7bf7062c6eaaf9acaf18efb753b70e4e"
server_key_version: "encryptionkey@0"
I20260925 15:13:42.555166 20095 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260925 15:13:42.558418 20110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:42.559616 20095 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 15:13:42.559789 20095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/minicluster-data/ts-1/wal
uuid: "bf7cab7961534d01af401cdd0fc4c5ff"
format_stamp: "Formatted at 2026-09-25 15:13:42 on dist-test-slave-p468"
server_key: "0af7098b75bd7b103f9af24efbfad020"
server_key_iv: "7bf7062c6eaaf9acaf18efb753b70e4e"
server_key_version: "encryptionkey@0"
I20260925 15:13:42.560015 20095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:13:42.594172 20095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:13:42.598526 20095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:13:42.598788 20095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:13:42.601449 20095 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:13:42.601549 20095 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:42.601680 20095 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:13:42.601758 20095 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 15:13:42 dist-test-slave-p468 krb5kdc[19839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349222, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.8.2@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM
I20260925 15:13:42.624567 19978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:42.602102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.2:43447 (local address 127.12.8.62:32861)
0925 15:13:42.602329 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:42.602337 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:42.603799 (+ 1462us) server_negotiation.cc:407] Connection header received
0925 15:13:42.605751 (+ 1952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:42.605756 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:42.605844 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:42.605953 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:42.608761 (+ 2808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:42.609838 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:42.612492 (+ 2654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:42.613170 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:42.613284 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:42.618015 (+ 4731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:42.618026 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:42.618033 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:42.618086 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:42.621472 (+ 3386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:42.622322 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:42.622327 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:42.622333 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:42.622454 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:42.622896 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:42.622901 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:42.622904 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:42.623170 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:42.623330 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:42.624014 (+ 684us) server_negotiation.cc:299] Negotiation successful
0925 15:13:42.624230 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260925 15:13:42.625545 20119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:42.602717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:32861 (local address 127.12.8.2:43447)
0925 15:13:42.603600 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:42.603678 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:42.605485 (+ 1807us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:42.606324 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:42.606340 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:42.607437 (+ 1097us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:13:42.608508 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:42.608528 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:42.610081 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:42.610085 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:42.612325 (+ 2240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:42.612344 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:42.613401 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:42.613412 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:42.613645 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:42.614373 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:13:42.614422 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:13:42.617725 (+ 3303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:13:42.621609 (+ 3884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:13:42.621619 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:13:42.621636 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:13:42.622148 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:13:42.622570 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:13:42.622575 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:13:42.622579 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:13:42.622768 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:13:42.623307 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:13:42.623314 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:13:42.623640 (+ 326us) client_negotiation.cc:769] Sending connection context
0925 15:13:42.623892 (+ 252us) client_negotiation.cc:240] Negotiation successful
0925 15:13:42.624392 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"thread_start_us":353,"threads_started":1}
I20260925 15:13:42.650106 20174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:42.631741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:32861 (local address 127.12.8.2:45871)
0925 15:13:42.635127 (+ 3386us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:42.635153 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:42.635326 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:42.635743 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:42.635749 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:42.636529 (+ 780us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:13:42.637037 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:42.637056 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:42.638677 (+ 1621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:42.638683 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:42.640983 (+ 2300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:42.640996 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:42.641960 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:42.641966 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:42.642175 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:42.642990 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:13:42.643019 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:13:42.643749 (+ 730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:13:42.647135 (+ 3386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:13:42.647143 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:13:42.647154 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:13:42.647800 (+ 646us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:13:42.648215 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:13:42.648222 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:13:42.648230 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:13:42.648407 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:13:42.649251 (+ 844us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:13:42.649259 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:13:42.649450 (+ 191us) client_negotiation.cc:769] Sending connection context
0925 15:13:42.649567 (+ 117us) client_negotiation.cc:240] Negotiation successful
0925 15:13:42.649785 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3190,"thread_start_us":2995,"threads_started":1}
I20260925 15:13:42.650516 19978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:42.632298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.2:45871 (local address 127.12.8.62:32861)
0925 15:13:42.632484 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:42.632490 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:42.635259 (+ 2769us) server_negotiation.cc:407] Connection header received
0925 15:13:42.635499 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:42.635507 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:42.635602 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:42.635720 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:42.637248 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:42.638489 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:42.641124 (+ 2635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:42.641759 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:42.641905 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:42.643938 (+ 2033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:42.643947 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:42.643954 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:42.643997 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:42.647001 (+ 3004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:42.647936 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:42.647943 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:42.647946 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:42.648093 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:42.648527 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:42.648530 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:42.648532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:42.649022 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:42.649175 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:42.649732 (+ 557us) server_negotiation.cc:299] Negotiation successful
0925 15:13:42.650029 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 15:13:42.657634 20095 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.2:36763
I20260925 15:13:42.657656 20229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.2:36763 every 8 connection(s)
I20260925 15:13:42.658892 20095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/minicluster-data/ts-1/data/info.pb
I20260925 15:13:42.660830 20230 heartbeater.cc:347] Connected to a master server at 127.12.8.62:32861
I20260925 15:13:42.661048 20230 heartbeater.cc:464] Registering TS with master...
I20260925 15:13:42.661473 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 20095
I20260925 15:13:42.661638 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/minicluster-data/ts-1/wal/instance
I20260925 15:13:42.661727 20230 heartbeater.cc:511] Master 127.12.8.62:32861 requested a full tablet report, sending...
I20260925 15:13:42.662122 12320 external_mini_cluster.cc:1428] Setting key 20dd23a15f97513a15b0d864d1d0fa0a
I20260925 15:13:42.663132 19884 ts_manager.cc:194] Registered new tserver with Master: bf7cab7961534d01af401cdd0fc4c5ff (127.12.8.2:36763)
I20260925 15:13:42.664065 19884 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.8.2@KRBTEST.COM'} at 127.12.8.2:43447
WARNING: no policy specified for oryx/127.12.8.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.8.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.8.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/krb5kdc/oryx_127.12.8.3.keytab.
Entry for principal oryx/127.12.8.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/krb5kdc/oryx_127.12.8.3.keytab.
I20260925 15:13:42.709230 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.3:0
--local_ip_for_outbound_sockets=127.12.8.3
--webserver_interface=127.12.8.3
--webserver_port=0
--tserver_master_addrs=127.12.8.62:32861
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/krb5kdc/oryx_127.12.8.3.keytab
--principal=oryx/127.12.8.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:35675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:13:43.032995 20234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:13:43.033262 20234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:13:43.033429 20234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:13:43.040560 20234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:13:43.040686 20234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:13:43.040838 20234 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 15:13:43.040961 20234 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.3
I20260925 15:13:43.050669 20234 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:35675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/krb5kdc/oryx_127.12.8.3.keytab
--principal=oryx/127.12.8.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.8.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.8.62:32861
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.3
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:13:43.051926 20234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:13:43.053205 20234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:13:43.066247 20240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:43.066216 20239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:43.066262 20242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:13:43.067440 20234 server_base.cc:1034] running on GCE node
I20260925 15:13:43.068017 20234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:13:43.068938 20234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:13:43.070163 20234 hybrid_clock.cc:648] HybridClock initialized: now 1790349223070104 us; error 31 us; skew 500 ppm
I20260925 15:13:43.070492 20234 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:13:43 dist-test-slave-p468 krb5kdc[19839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349223, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.8.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:13:43.073534 20234 init.cc:377] Logged in from keytab as oryx/127.12.8.3@KRBTEST.COM (short username oryx)
I20260925 15:13:43.074910 20234 webserver.cc:460] Webserver started at http://127.12.8.3:36023/ using document root <none> and password file <none>
I20260925 15:13:43.075507 20234 fs_manager.cc:359] Metadata directory not provided
I20260925 15:13:43.075587 20234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:13:43.075851 20234 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:13:43.078558 20234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/minicluster-data/ts-2/data/instance:
uuid: "b7427e659a2b4fff8dc2e523e45d0718"
format_stamp: "Formatted at 2026-09-25 15:13:43 on dist-test-slave-p468"
server_key: "db827a5ca81afe38088d0ced9e7a8841"
server_key_iv: "59eda174b93c66c9e17c93ce7e795e89"
server_key_version: "encryptionkey@0"
I20260925 15:13:43.079273 20234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/minicluster-data/ts-2/wal/instance:
uuid: "b7427e659a2b4fff8dc2e523e45d0718"
format_stamp: "Formatted at 2026-09-25 15:13:43 on dist-test-slave-p468"
server_key: "db827a5ca81afe38088d0ced9e7a8841"
server_key_iv: "59eda174b93c66c9e17c93ce7e795e89"
server_key_version: "encryptionkey@0"
I20260925 15:13:43.084159 20234 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 15:13:43.087407 20249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:43.088650 20234 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 15:13:43.088809 20234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/minicluster-data/ts-2/wal
uuid: "b7427e659a2b4fff8dc2e523e45d0718"
format_stamp: "Formatted at 2026-09-25 15:13:43 on dist-test-slave-p468"
server_key: "db827a5ca81afe38088d0ced9e7a8841"
server_key_iv: "59eda174b93c66c9e17c93ce7e795e89"
server_key_version: "encryptionkey@0"
I20260925 15:13:43.089040 20234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:13:43.103950 20234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:13:43.108140 20234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:13:43.108402 20234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:13:43.110755 20234 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:13:43.110867 20234 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:43.111013 20234 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:13:43.111089 20234 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 15:13:43 dist-test-slave-p468 krb5kdc[19839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349223, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.8.3@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM
I20260925 15:13:43.137238 19978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:43.111518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.3:52821 (local address 127.12.8.62:32861)
0925 15:13:43.111741 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:43.111751 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:43.113017 (+ 1266us) server_negotiation.cc:407] Connection header received
0925 15:13:43.115273 (+ 2256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:43.115283 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:43.115388 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:43.115504 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:43.117557 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:43.118616 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:43.121169 (+ 2553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:43.121558 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:43.121686 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:43.127223 (+ 5537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:43.127231 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:43.127237 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:43.127283 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:43.131311 (+ 4028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:43.132160 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:43.132168 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:43.132175 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:43.132378 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:43.133055 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:43.133063 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:43.133069 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:43.133538 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:43.136211 (+ 2673us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:43.136420 (+ 209us) server_negotiation.cc:299] Negotiation successful
0925 15:13:43.136722 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 15:13:43.138130 20258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:43.111989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:32861 (local address 127.12.8.3:52821)
0925 15:13:43.112856 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:43.112897 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:43.114986 (+ 2089us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:43.115735 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:43.115748 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:43.116553 (+ 805us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:13:43.117374 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:43.117404 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:43.118798 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:43.118802 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:43.120976 (+ 2174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:43.120991 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:43.121801 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:43.121810 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:43.122160 (+ 350us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:43.123050 (+ 890us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:13:43.123085 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:13:43.126936 (+ 3851us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:13:43.131523 (+ 4587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:13:43.131534 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:13:43.131554 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:13:43.132021 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:13:43.132540 (+ 519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:13:43.132549 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:13:43.132554 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:13:43.132858 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:13:43.133716 (+ 858us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:13:43.133728 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:13:43.134271 (+ 543us) client_negotiation.cc:769] Sending connection context
0925 15:13:43.134629 (+ 358us) client_negotiation.cc:240] Negotiation successful
0925 15:13:43.135305 (+ 676us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"thread_start_us":400,"threads_started":1}
I20260925 15:13:43.141749 20091 heartbeater.cc:503] Master 127.12.8.62:32861 was elected leader, sending a full tablet report...
I20260925 15:13:43.167543 19978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:43.152815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.3:36823 (local address 127.12.8.62:32861)
0925 15:13:43.152994 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:43.153000 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:43.153776 (+ 776us) server_negotiation.cc:407] Connection header received
0925 15:13:43.153965 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:43.153969 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:43.154026 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:43.154105 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:43.155672 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:43.156503 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:43.158595 (+ 2092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:43.159212 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:43.159353 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:43.161130 (+ 1777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:43.161134 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:43.161138 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:43.161165 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:43.164583 (+ 3418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:43.165391 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:43.165399 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:43.165405 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:43.165594 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:43.165958 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:43.165966 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:43.165970 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:43.166386 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:43.166554 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:43.166958 (+ 404us) server_negotiation.cc:299] Negotiation successful
0925 15:13:43.167181 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 15:13:43.167733 20341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:43.152877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:32861 (local address 127.12.8.3:36823)
0925 15:13:43.153651 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:43.153675 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:43.153845 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:43.154167 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:43.154174 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:43.155010 (+ 836us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:13:43.155521 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:43.155534 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:43.156693 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:43.156701 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:43.158382 (+ 1681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:43.158402 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:43.159414 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:43.159421 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:43.159578 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:43.160236 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:13:43.160252 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:13:43.160940 (+ 688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:13:43.164715 (+ 3775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:13:43.164718 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:13:43.164721 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:13:43.165239 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:13:43.165713 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:13:43.165716 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:13:43.165718 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:13:43.165831 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:13:43.166505 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:13:43.166510 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:13:43.166683 (+ 173us) client_negotiation.cc:769] Sending connection context
0925 15:13:43.166837 (+ 154us) client_negotiation.cc:240] Negotiation successful
0925 15:13:43.167193 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":352,"threads_started":1}
I20260925 15:13:43.168569 20234 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.3:43529
I20260925 15:13:43.168589 20368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.3:43529 every 8 connection(s)
I20260925 15:13:43.170089 20234 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/minicluster-data/ts-2/data/info.pb
I20260925 15:13:43.172084 20369 heartbeater.cc:347] Connected to a master server at 127.12.8.62:32861
I20260925 15:13:43.172259 20369 heartbeater.cc:464] Registering TS with master...
I20260925 15:13:43.172888 20369 heartbeater.cc:511] Master 127.12.8.62:32861 requested a full tablet report, sending...
I20260925 15:13:43.174016 19884 ts_manager.cc:194] Registered new tserver with Master: b7427e659a2b4fff8dc2e523e45d0718 (127.12.8.3:43529)
I20260925 15:13:43.174867 19884 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.8.3@KRBTEST.COM'} at 127.12.8.3:52821
I20260925 15:13:43.178293 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 20234
I20260925 15:13:43.178431 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/minicluster-data/ts-2/wal/instance
I20260925 15:13:43.178804 12320 external_mini_cluster.cc:1428] Setting key f1a850768230d41222a726c7b450a26b
I20260925 15:13:43.181294 12320 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 15:13:43 dist-test-slave-p468 krb5kdc[19839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349223, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 15:13:43 dist-test-slave-p468 krb5kdc[19839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:13:43 dist-test-slave-p468 krb5kdc[19839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 15:13:43.229236 19978 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:13:43.219053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58200 (local address 127.12.8.62:32861)
0925 15:13:43.219275 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:43.219283 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:43.219513 (+ 230us) server_negotiation.cc:407] Connection header received
0925 15:13:43.219715 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:43.219720 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:43.219791 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:43.219900 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:43.221115 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:43.221825 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:43.223116 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:43.223520 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:43.223646 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:43.228926 (+ 5280us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58200: BlockingRecv error: recv got EOF from 127.0.0.1:58200 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 25 15:13:43 dist-test-slave-p468 krb5kdc[19839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349223, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM
I20260925 15:13:43.243923 19978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:43.231870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58216 (local address 127.12.8.62:32861)
0925 15:13:43.232023 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:43.232027 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:43.232371 (+ 344us) server_negotiation.cc:407] Connection header received
0925 15:13:43.232497 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:43.232500 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:43.232571 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:43.232655 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:43.233729 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:43.234477 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:43.235601 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:43.236038 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:43.236128 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:43.239446 (+ 3318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:43.239450 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:43.239454 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:43.239483 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:43.241910 (+ 2427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:43.242555 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:43.242558 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:43.242561 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:43.242670 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:43.242952 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:43.242955 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:43.242957 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:43.243195 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:43.243408 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:43.243556 (+ 148us) server_negotiation.cc:299] Negotiation successful
0925 15:13:43.243703 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 15:13:43.247521 19884 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58216:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:13:43.254151 19884 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:13:43.283934 20390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:43.266507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56062 (local address 127.12.8.1:39179)
0925 15:13:43.275824 (+ 9317us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:43.275833 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:43.275868 (+ 35us) server_negotiation.cc:407] Connection header received
0925 15:13:43.275988 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:43.276000 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:43.276320 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:43.276563 (+ 243us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:13:43.277795 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:43.279454 (+ 1659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:43.281253 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:43.282749 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:43.282871 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:43.283096 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:43.283455 (+ 359us) server_negotiation.cc:299] Negotiation successful
0925 15:13:43.283620 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9134,"thread_start_us":9012,"threads_started":1}
I20260925 15:13:43.284015 20389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:43.266021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.1:39179 (local address 127.0.0.1:56062)
0925 15:13:43.275518 (+ 9497us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:43.275539 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:43.275664 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:43.276849 (+ 1185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:43.276860 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:43.276899 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:13:43.277451 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:43.277467 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:43.279586 (+ 2119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:43.279591 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:43.281101 (+ 1510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:43.281116 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:43.282941 (+ 1825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:43.282947 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:43.283185 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:43.283255 (+ 70us) client_negotiation.cc:769] Sending connection context
0925 15:13:43.283378 (+ 123us) client_negotiation.cc:240] Negotiation successful
0925 15:13:43.283570 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9361,"mutex_wait_us":24,"thread_start_us":9215,"threads_started":1}
I20260925 15:13:43.285470 20392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:43.267865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33624 (local address 127.12.8.2:36763)
0925 15:13:43.276142 (+ 8277us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:43.276154 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:43.277897 (+ 1743us) server_negotiation.cc:407] Connection header received
0925 15:13:43.278074 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:43.278079 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:43.278316 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:43.278527 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:13:43.279173 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:43.280597 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:43.283242 (+ 2645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:43.284515 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:43.284656 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:43.284850 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:43.285058 (+ 208us) server_negotiation.cc:299] Negotiation successful
0925 15:13:43.285202 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8090,"thread_start_us":7980,"threads_started":1}
I20260925 15:13:43.285480 20391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:43.267555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.2:36763 (local address 127.0.0.1:33624)
0925 15:13:43.277777 (+ 10222us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:43.277791 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:43.277917 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:43.278651 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:43.278656 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:43.278680 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:13:43.279005 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:43.279016 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:43.280772 (+ 1756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:43.280777 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:43.283042 (+ 2265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:43.283055 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:43.284702 (+ 1647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:43.284709 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:43.284861 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:43.284897 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 15:13:43.285004 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 15:13:43.285124 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10145,"mutex_wait_us":54,"thread_start_us":9961,"threads_started":1}
I20260925 15:13:43.286689 20387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:43.265333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.3:43529 (local address 127.0.0.1:56158)
0925 15:13:43.267172 (+ 1839us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:43.267206 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:43.267367 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:43.279145 (+ 11778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:43.279150 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:43.279176 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:13:43.279576 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:43.279585 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:43.281525 (+ 1940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:43.281535 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:43.283648 (+ 2113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:43.283661 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:43.285890 (+ 2229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:43.285896 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:43.286067 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:43.286103 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 15:13:43.286210 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 15:13:43.286333 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1607,"mutex_wait_us":46,"thread_start_us":1431,"threads_started":1}
I20260925 15:13:43.286737 20388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:43.265671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56158 (local address 127.12.8.3:43529)
0925 15:13:43.278731 (+ 13060us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:43.278737 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:43.278760 (+ 23us) server_negotiation.cc:407] Connection header received
0925 15:13:43.278831 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:43.278836 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:43.278981 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:43.279152 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:13:43.279798 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:43.281362 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:43.283829 (+ 2467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:43.285701 (+ 1872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:43.285837 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:43.286041 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:43.286292 (+ 251us) server_negotiation.cc:299] Negotiation successful
0925 15:13:43.286438 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12894,"thread_start_us":12713,"threads_started":1}
I20260925 15:13:43.288158 20021 tablet_service.cc:1467] Processing CreateTablet for tablet e3527aa97b18433bbf01269bf4f25849 (DEFAULT_TABLE table=test-table [id=51ed340122e140e48bce487161435930]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:13:43.289587 20021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3527aa97b18433bbf01269bf4f25849. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:13:43.289216 20160 tablet_service.cc:1467] Processing CreateTablet for tablet e3527aa97b18433bbf01269bf4f25849 (DEFAULT_TABLE table=test-table [id=51ed340122e140e48bce487161435930]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:13:43.290508 20160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3527aa97b18433bbf01269bf4f25849. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:13:43.295706 20299 tablet_service.cc:1467] Processing CreateTablet for tablet e3527aa97b18433bbf01269bf4f25849 (DEFAULT_TABLE table=test-table [id=51ed340122e140e48bce487161435930]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:13:43.296936 20299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3527aa97b18433bbf01269bf4f25849. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:13:43.306100 20393 tablet_bootstrap.cc:492] T e3527aa97b18433bbf01269bf4f25849 P c2e6c5753e2546d9bc00393d25672551: Bootstrap starting.
I20260925 15:13:43.308480 20395 tablet_bootstrap.cc:492] T e3527aa97b18433bbf01269bf4f25849 P b7427e659a2b4fff8dc2e523e45d0718: Bootstrap starting.
I20260925 15:13:43.309585 20393 tablet_bootstrap.cc:654] T e3527aa97b18433bbf01269bf4f25849 P c2e6c5753e2546d9bc00393d25672551: Neither blocks nor log segments found. Creating new log.
I20260925 15:13:43.304582 20394 tablet_bootstrap.cc:492] T e3527aa97b18433bbf01269bf4f25849 P bf7cab7961534d01af401cdd0fc4c5ff: Bootstrap starting.
I20260925 15:13:43.310794 20393 log.cc:824] T e3527aa97b18433bbf01269bf4f25849 P c2e6c5753e2546d9bc00393d25672551: Log is configured to *not* fsync() on all Append() calls
I20260925 15:13:43.311252 20395 tablet_bootstrap.cc:654] T e3527aa97b18433bbf01269bf4f25849 P b7427e659a2b4fff8dc2e523e45d0718: Neither blocks nor log segments found. Creating new log.
I20260925 15:13:43.312448 20395 log.cc:824] T e3527aa97b18433bbf01269bf4f25849 P b7427e659a2b4fff8dc2e523e45d0718: Log is configured to *not* fsync() on all Append() calls
I20260925 15:13:43.313287 20394 tablet_bootstrap.cc:654] T e3527aa97b18433bbf01269bf4f25849 P bf7cab7961534d01af401cdd0fc4c5ff: Neither blocks nor log segments found. Creating new log.
I20260925 15:13:43.314031 20393 tablet_bootstrap.cc:492] T e3527aa97b18433bbf01269bf4f25849 P c2e6c5753e2546d9bc00393d25672551: No bootstrap required, opened a new log
I20260925 15:13:43.314337 20393 ts_tablet_manager.cc:1397] T e3527aa97b18433bbf01269bf4f25849 P c2e6c5753e2546d9bc00393d25672551: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.003s
I20260925 15:13:43.314489 20394 log.cc:824] T e3527aa97b18433bbf01269bf4f25849 P bf7cab7961534d01af401cdd0fc4c5ff: Log is configured to *not* fsync() on all Append() calls
I20260925 15:13:43.315102 20395 tablet_bootstrap.cc:492] T e3527aa97b18433bbf01269bf4f25849 P b7427e659a2b4fff8dc2e523e45d0718: No bootstrap required, opened a new log
I20260925 15:13:43.315366 20395 ts_tablet_manager.cc:1397] T e3527aa97b18433bbf01269bf4f25849 P b7427e659a2b4fff8dc2e523e45d0718: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.001s
I20260925 15:13:43.317350 20394 tablet_bootstrap.cc:492] T e3527aa97b18433bbf01269bf4f25849 P bf7cab7961534d01af401cdd0fc4c5ff: No bootstrap required, opened a new log
I20260925 15:13:43.317595 20394 ts_tablet_manager.cc:1397] T e3527aa97b18433bbf01269bf4f25849 P bf7cab7961534d01af401cdd0fc4c5ff: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.000s
I20260925 15:13:43.322528 20395 raft_consensus.cc:348] T e3527aa97b18433bbf01269bf4f25849 P b7427e659a2b4fff8dc2e523e45d0718 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7427e659a2b4fff8dc2e523e45d0718" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43529 } } peers { permanent_uuid: "bf7cab7961534d01af401cdd0fc4c5ff" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 36763 } } peers { permanent_uuid: "c2e6c5753e2546d9bc00393d25672551" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 39179 } }
I20260925 15:13:43.322883 20395 raft_consensus.cc:374] T e3527aa97b18433bbf01269bf4f25849 P b7427e659a2b4fff8dc2e523e45d0718 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:13:43.323009 20395 raft_consensus.cc:729] T e3527aa97b18433bbf01269bf4f25849 P b7427e659a2b4fff8dc2e523e45d0718 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7427e659a2b4fff8dc2e523e45d0718, State: Initialized, Role: FOLLOWER
I20260925 15:13:43.323576 20395 consensus_queue.cc:260] T e3527aa97b18433bbf01269bf4f25849 P b7427e659a2b4fff8dc2e523e45d0718 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7427e659a2b4fff8dc2e523e45d0718" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43529 } } peers { permanent_uuid: "bf7cab7961534d01af401cdd0fc4c5ff" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 36763 } } peers { permanent_uuid: "c2e6c5753e2546d9bc00393d25672551" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 39179 } }
I20260925 15:13:43.324615 20369 heartbeater.cc:503] Master 127.12.8.62:32861 was elected leader, sending a full tablet report...
I20260925 15:13:43.324851 20393 raft_consensus.cc:348] T e3527aa97b18433bbf01269bf4f25849 P c2e6c5753e2546d9bc00393d25672551 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7427e659a2b4fff8dc2e523e45d0718" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43529 } } peers { permanent_uuid: "bf7cab7961534d01af401cdd0fc4c5ff" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 36763 } } peers { permanent_uuid: "c2e6c5753e2546d9bc00393d25672551" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 39179 } }
I20260925 15:13:43.325224 20393 raft_consensus.cc:374] T e3527aa97b18433bbf01269bf4f25849 P c2e6c5753e2546d9bc00393d25672551 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:13:43.325233 20395 ts_tablet_manager.cc:1428] T e3527aa97b18433bbf01269bf4f25849 P b7427e659a2b4fff8dc2e523e45d0718: Time spent starting tablet: real 0.010s user 0.003s sys 0.006s
I20260925 15:13:43.325299 20393 raft_consensus.cc:729] T e3527aa97b18433bbf01269bf4f25849 P c2e6c5753e2546d9bc00393d25672551 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2e6c5753e2546d9bc00393d25672551, State: Initialized, Role: FOLLOWER
I20260925 15:13:43.325837 20393 consensus_queue.cc:260] T e3527aa97b18433bbf01269bf4f25849 P c2e6c5753e2546d9bc00393d25672551 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7427e659a2b4fff8dc2e523e45d0718" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43529 } } peers { permanent_uuid: "bf7cab7961534d01af401cdd0fc4c5ff" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 36763 } } peers { permanent_uuid: "c2e6c5753e2546d9bc00393d25672551" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 39179 } }
I20260925 15:13:43.327402 20393 ts_tablet_manager.cc:1428] T e3527aa97b18433bbf01269bf4f25849 P c2e6c5753e2546d9bc00393d25672551: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260925 15:13:43.327888 20394 raft_consensus.cc:348] T e3527aa97b18433bbf01269bf4f25849 P bf7cab7961534d01af401cdd0fc4c5ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7427e659a2b4fff8dc2e523e45d0718" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43529 } } peers { permanent_uuid: "bf7cab7961534d01af401cdd0fc4c5ff" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 36763 } } peers { permanent_uuid: "c2e6c5753e2546d9bc00393d25672551" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 39179 } }
I20260925 15:13:43.328260 20394 raft_consensus.cc:374] T e3527aa97b18433bbf01269bf4f25849 P bf7cab7961534d01af401cdd0fc4c5ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:13:43.328445 20394 raft_consensus.cc:729] T e3527aa97b18433bbf01269bf4f25849 P bf7cab7961534d01af401cdd0fc4c5ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf7cab7961534d01af401cdd0fc4c5ff, State: Initialized, Role: FOLLOWER
I20260925 15:13:43.329190 20394 consensus_queue.cc:260] T e3527aa97b18433bbf01269bf4f25849 P bf7cab7961534d01af401cdd0fc4c5ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7427e659a2b4fff8dc2e523e45d0718" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43529 } } peers { permanent_uuid: "bf7cab7961534d01af401cdd0fc4c5ff" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 36763 } } peers { permanent_uuid: "c2e6c5753e2546d9bc00393d25672551" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 39179 } }
I20260925 15:13:43.330317 20230 heartbeater.cc:503] Master 127.12.8.62:32861 was elected leader, sending a full tablet report...
I20260925 15:13:43.330950 20394 ts_tablet_manager.cc:1428] T e3527aa97b18433bbf01269bf4f25849 P bf7cab7961534d01af401cdd0fc4c5ff: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
W20260925 15:13:43.384269 20092 tablet.cc:2369] T e3527aa97b18433bbf01269bf4f25849 P c2e6c5753e2546d9bc00393d25672551: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:13:43.406599 20401 raft_consensus.cc:482] T e3527aa97b18433bbf01269bf4f25849 P bf7cab7961534d01af401cdd0fc4c5ff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:13:43.406875 20401 raft_consensus.cc:504] T e3527aa97b18433bbf01269bf4f25849 P bf7cab7961534d01af401cdd0fc4c5ff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7427e659a2b4fff8dc2e523e45d0718" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43529 } } peers { permanent_uuid: "bf7cab7961534d01af401cdd0fc4c5ff" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 36763 } } peers { permanent_uuid: "c2e6c5753e2546d9bc00393d25672551" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 39179 } }
I20260925 15:13:43.408538 20401 leader_election.cc:288] T e3527aa97b18433bbf01269bf4f25849 P bf7cab7961534d01af401cdd0fc4c5ff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b7427e659a2b4fff8dc2e523e45d0718 (127.12.8.3:43529), c2e6c5753e2546d9bc00393d25672551 (127.12.8.1:39179)
W20260925 15:13:43.410830 20231 tablet.cc:2369] T e3527aa97b18433bbf01269bf4f25849 P bf7cab7961534d01af401cdd0fc4c5ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:13:43.415495 20388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:43.409060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.2:45303 (local address 127.12.8.3:43529)
0925 15:13:43.409278 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:43.409287 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:43.409848 (+ 561us) server_negotiation.cc:407] Connection header received
0925 15:13:43.409998 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:43.410006 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:43.410082 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:43.410203 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:13:43.411112 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:43.411871 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:43.413792 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:43.414587 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:43.414732 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:43.414843 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:43.415118 (+ 275us) server_negotiation.cc:299] Negotiation successful
0925 15:13:43.415234 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 15:13:43.415549 20390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:43.409061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.2:40263 (local address 127.12.8.1:39179)
0925 15:13:43.409271 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:43.409280 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:43.409762 (+ 482us) server_negotiation.cc:407] Connection header received
0925 15:13:43.409989 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:43.409994 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:43.410082 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:43.410216 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:13:43.411022 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:43.411943 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:43.413818 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:43.414655 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:43.414809 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:43.414931 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:43.415181 (+ 250us) server_negotiation.cc:299] Negotiation successful
0925 15:13:43.415302 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 15:13:43.415668 20403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:43.408966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.3:43529 (local address 127.12.8.2:45303)
0925 15:13:43.409759 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:43.409774 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:43.409880 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:43.410227 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:43.410233 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:43.410270 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:13:43.410904 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:43.410916 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:43.411991 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:43.411999 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:43.413649 (+ 1650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:43.413661 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:43.414800 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:43.414806 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:43.414965 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:43.414988 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 15:13:43.415084 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 15:13:43.415209 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":673,"mutex_wait_us":38,"spinlock_wait_cycles":896,"thread_start_us":490,"threads_started":1}
I20260925 15:13:43.415715 20402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:43.408820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.1:39179 (local address 127.12.8.2:40263)
0925 15:13:43.409644 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:43.409664 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:43.409827 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:43.410227 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:43.410233 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:43.410276 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:13:43.410846 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:43.410866 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:43.412084 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:43.412092 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:43.413666 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:43.413679 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:43.414873 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:43.414886 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:43.415034 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:43.415056 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 15:13:43.415157 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 15:13:43.415277 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"mutex_wait_us":67,"spinlock_wait_cycles":6656,"thread_start_us":466,"threads_started":1}
I20260925 15:13:43.416725 20046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3527aa97b18433bbf01269bf4f25849" candidate_uuid: "bf7cab7961534d01af401cdd0fc4c5ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2e6c5753e2546d9bc00393d25672551" is_pre_election: true
I20260925 15:13:43.417157 20046 raft_consensus.cc:2454] T e3527aa97b18433bbf01269bf4f25849 P c2e6c5753e2546d9bc00393d25672551 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf7cab7961534d01af401cdd0fc4c5ff in term 0.
I20260925 15:13:43.417924 20113 leader_election.cc:302] T e3527aa97b18433bbf01269bf4f25849 P bf7cab7961534d01af401cdd0fc4c5ff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bf7cab7961534d01af401cdd0fc4c5ff, c2e6c5753e2546d9bc00393d25672551; no voters:
I20260925 15:13:43.417991 20319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3527aa97b18433bbf01269bf4f25849" candidate_uuid: "bf7cab7961534d01af401cdd0fc4c5ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7427e659a2b4fff8dc2e523e45d0718" is_pre_election: true
I20260925 15:13:43.418334 20401 raft_consensus.cc:2781] T e3527aa97b18433bbf01269bf4f25849 P bf7cab7961534d01af401cdd0fc4c5ff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:13:43.418463 20401 raft_consensus.cc:482] T e3527aa97b18433bbf01269bf4f25849 P bf7cab7961534d01af401cdd0fc4c5ff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:13:43.418552 20401 raft_consensus.cc:3037] T e3527aa97b18433bbf01269bf4f25849 P bf7cab7961534d01af401cdd0fc4c5ff [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:13:43.418534 20319 raft_consensus.cc:2454] T e3527aa97b18433bbf01269bf4f25849 P b7427e659a2b4fff8dc2e523e45d0718 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf7cab7961534d01af401cdd0fc4c5ff in term 0.
I20260925 15:13:43.420840 20401 raft_consensus.cc:504] T e3527aa97b18433bbf01269bf4f25849 P bf7cab7961534d01af401cdd0fc4c5ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7427e659a2b4fff8dc2e523e45d0718" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43529 } } peers { permanent_uuid: "bf7cab7961534d01af401cdd0fc4c5ff" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 36763 } } peers { permanent_uuid: "c2e6c5753e2546d9bc00393d25672551" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 39179 } }
I20260925 15:13:43.421403 20401 leader_election.cc:288] T e3527aa97b18433bbf01269bf4f25849 P bf7cab7961534d01af401cdd0fc4c5ff [CANDIDATE]: Term 1 election: Requested vote from peers b7427e659a2b4fff8dc2e523e45d0718 (127.12.8.3:43529), c2e6c5753e2546d9bc00393d25672551 (127.12.8.1:39179)
I20260925 15:13:43.421777 20319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3527aa97b18433bbf01269bf4f25849" candidate_uuid: "bf7cab7961534d01af401cdd0fc4c5ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7427e659a2b4fff8dc2e523e45d0718"
I20260925 15:13:43.421887 20046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3527aa97b18433bbf01269bf4f25849" candidate_uuid: "bf7cab7961534d01af401cdd0fc4c5ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2e6c5753e2546d9bc00393d25672551"
I20260925 15:13:43.421980 20319 raft_consensus.cc:3037] T e3527aa97b18433bbf01269bf4f25849 P b7427e659a2b4fff8dc2e523e45d0718 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:13:43.422081 20046 raft_consensus.cc:3037] T e3527aa97b18433bbf01269bf4f25849 P c2e6c5753e2546d9bc00393d25672551 [term 0 FOLLOWER]: Advancing to term 1
W20260925 15:13:43.422093 20370 tablet.cc:2369] T e3527aa97b18433bbf01269bf4f25849 P b7427e659a2b4fff8dc2e523e45d0718: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:13:43.425189 20319 raft_consensus.cc:2454] T e3527aa97b18433bbf01269bf4f25849 P b7427e659a2b4fff8dc2e523e45d0718 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf7cab7961534d01af401cdd0fc4c5ff in term 1.
I20260925 15:13:43.425189 20046 raft_consensus.cc:2454] T e3527aa97b18433bbf01269bf4f25849 P c2e6c5753e2546d9bc00393d25672551 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf7cab7961534d01af401cdd0fc4c5ff in term 1.
I20260925 15:13:43.425562 20113 leader_election.cc:302] T e3527aa97b18433bbf01269bf4f25849 P bf7cab7961534d01af401cdd0fc4c5ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bf7cab7961534d01af401cdd0fc4c5ff, c2e6c5753e2546d9bc00393d25672551; no voters:
I20260925 15:13:43.425846 20401 raft_consensus.cc:2781] T e3527aa97b18433bbf01269bf4f25849 P bf7cab7961534d01af401cdd0fc4c5ff [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:13:43.426318 20401 raft_consensus.cc:686] T e3527aa97b18433bbf01269bf4f25849 P bf7cab7961534d01af401cdd0fc4c5ff [term 1 LEADER]: Becoming Leader. State: Replica: bf7cab7961534d01af401cdd0fc4c5ff, State: Running, Role: LEADER
I20260925 15:13:43.426752 20401 consensus_queue.cc:237] T e3527aa97b18433bbf01269bf4f25849 P bf7cab7961534d01af401cdd0fc4c5ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7427e659a2b4fff8dc2e523e45d0718" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43529 } } peers { permanent_uuid: "bf7cab7961534d01af401cdd0fc4c5ff" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 36763 } } peers { permanent_uuid: "c2e6c5753e2546d9bc00393d25672551" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 39179 } }
I20260925 15:13:43.430964 19883 catalog_manager.cc:5495] T e3527aa97b18433bbf01269bf4f25849 P bf7cab7961534d01af401cdd0fc4c5ff reported cstate change: term changed from 0 to 1, leader changed from <none> to bf7cab7961534d01af401cdd0fc4c5ff (127.12.8.2). New cstate: current_term: 1 leader_uuid: "bf7cab7961534d01af401cdd0fc4c5ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7427e659a2b4fff8dc2e523e45d0718" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf7cab7961534d01af401cdd0fc4c5ff" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 36763 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2e6c5753e2546d9bc00393d25672551" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 39179 } health_report { overall_health: UNKNOWN } } }
I20260925 15:13:43.467128 20392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:43.461804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35694 (local address 127.12.8.2:36763)
0925 15:13:43.462009 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:43.462014 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:43.462034 (+ 20us) server_negotiation.cc:407] Connection header received
0925 15:13:43.462300 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:43.462304 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:43.462381 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:43.462531 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:13:43.462861 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:43.463679 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:43.465024 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:43.465443 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:43.465571 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:43.466005 (+ 434us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:13:43.466579 (+ 574us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:13:43.466735 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:43.466793 (+ 58us) server_negotiation.cc:299] Negotiation successful
0925 15:13:43.466868 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260925 15:13:43.474869 20046 raft_consensus.cc:1261] T e3527aa97b18433bbf01269bf4f25849 P c2e6c5753e2546d9bc00393d25672551 [term 1 FOLLOWER]: Refusing update from remote peer bf7cab7961534d01af401cdd0fc4c5ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:13:43.474866 20319 raft_consensus.cc:1261] T e3527aa97b18433bbf01269bf4f25849 P b7427e659a2b4fff8dc2e523e45d0718 [term 1 FOLLOWER]: Refusing update from remote peer bf7cab7961534d01af401cdd0fc4c5ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:13:43.475802 20404 consensus_queue.cc:1035] T e3527aa97b18433bbf01269bf4f25849 P bf7cab7961534d01af401cdd0fc4c5ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7427e659a2b4fff8dc2e523e45d0718" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:13:43.476168 20401 consensus_queue.cc:1035] T e3527aa97b18433bbf01269bf4f25849 P bf7cab7961534d01af401cdd0fc4c5ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2e6c5753e2546d9bc00393d25672551" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 39179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:13:43.489313 20408 mvcc.cc:205] Tried to move back new op lower bound from 7333270419343798272 to 7333270419159740416. Current Snapshot: MvccSnapshot[applied={T|T < 7333270419343798272}]
I20260925 15:13:43.499138 19883 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58216:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:13:43.499439 19883 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58216:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:13:43.502967 19883 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3312e9eb760f4482923279ed63ae02a8: Sending DeleteTablet for 3 replicas of tablet e3527aa97b18433bbf01269bf4f25849
I20260925 15:13:43.504016 20299 tablet_service.cc:1514] Processing DeleteTablet for tablet e3527aa97b18433bbf01269bf4f25849 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:13:43 UTC) from {username='oryx', principal='oryx/127.12.8.62@KRBTEST.COM'} at 127.0.0.1:56158
I20260925 15:13:43.504076 20021 tablet_service.cc:1514] Processing DeleteTablet for tablet e3527aa97b18433bbf01269bf4f25849 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:13:43 UTC) from {username='oryx', principal='oryx/127.12.8.62@KRBTEST.COM'} at 127.0.0.1:56062
I20260925 15:13:43.504894 20416 tablet_replica.cc:330] T e3527aa97b18433bbf01269bf4f25849 P b7427e659a2b4fff8dc2e523e45d0718: stopping tablet replica
I20260925 15:13:43.505224 20160 tablet_service.cc:1514] Processing DeleteTablet for tablet e3527aa97b18433bbf01269bf4f25849 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:13:43 UTC) from {username='oryx', principal='oryx/127.12.8.62@KRBTEST.COM'} at 127.0.0.1:33624
I20260925 15:13:43.505354 20416 raft_consensus.cc:2229] T e3527aa97b18433bbf01269bf4f25849 P b7427e659a2b4fff8dc2e523e45d0718 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:13:43.505991 20416 raft_consensus.cc:2258] T e3527aa97b18433bbf01269bf4f25849 P b7427e659a2b4fff8dc2e523e45d0718 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:13:43.505982 19883 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58216:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:13:43.506671 19883 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:13:43.508484 20416 ts_tablet_manager.cc:1905] T e3527aa97b18433bbf01269bf4f25849 P b7427e659a2b4fff8dc2e523e45d0718: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:13:43.511132 20418 tablet_replica.cc:330] T e3527aa97b18433bbf01269bf4f25849 P bf7cab7961534d01af401cdd0fc4c5ff: stopping tablet replica
I20260925 15:13:43.511533 20418 raft_consensus.cc:2229] T e3527aa97b18433bbf01269bf4f25849 P bf7cab7961534d01af401cdd0fc4c5ff [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:13:43.512197 20418 raft_consensus.cc:2258] T e3527aa97b18433bbf01269bf4f25849 P bf7cab7961534d01af401cdd0fc4c5ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:13:43.514147 20418 ts_tablet_manager.cc:1905] T e3527aa97b18433bbf01269bf4f25849 P bf7cab7961534d01af401cdd0fc4c5ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:13:43.514871 20020 tablet_service.cc:1467] Processing CreateTablet for tablet e06b6369212d4277aa3452c4a11865cb (DEFAULT_TABLE table=test-table [id=c20cbef31ff7414fb90de53e1ae98bc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:13:43.515388 20020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e06b6369212d4277aa3452c4a11865cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:13:43.516824 20416 ts_tablet_manager.cc:1918] T e3527aa97b18433bbf01269bf4f25849 P b7427e659a2b4fff8dc2e523e45d0718: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:13:43.517032 20416 log.cc:1196] T e3527aa97b18433bbf01269bf4f25849 P b7427e659a2b4fff8dc2e523e45d0718: Deleting WAL directory at /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/minicluster-data/ts-2/wal/wals/e3527aa97b18433bbf01269bf4f25849
I20260925 15:13:43.517946 20416 ts_tablet_manager.cc:1939] T e3527aa97b18433bbf01269bf4f25849 P b7427e659a2b4fff8dc2e523e45d0718: Deleting consensus metadata
I20260925 15:13:43.518728 20299 tablet_service.cc:1467] Processing CreateTablet for tablet e06b6369212d4277aa3452c4a11865cb (DEFAULT_TABLE table=test-table [id=c20cbef31ff7414fb90de53e1ae98bc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:13:43.519155 20299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e06b6369212d4277aa3452c4a11865cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:13:43.519897 19869 catalog_manager.cc:4841] TS b7427e659a2b4fff8dc2e523e45d0718 (127.12.8.3:43529): tablet e3527aa97b18433bbf01269bf4f25849 (table test-table [id=51ed340122e140e48bce487161435930]) successfully deleted
I20260925 15:13:43.521272 20418 ts_tablet_manager.cc:1918] T e3527aa97b18433bbf01269bf4f25849 P bf7cab7961534d01af401cdd0fc4c5ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:13:43.521456 20418 log.cc:1196] T e3527aa97b18433bbf01269bf4f25849 P bf7cab7961534d01af401cdd0fc4c5ff: Deleting WAL directory at /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/minicluster-data/ts-1/wal/wals/e3527aa97b18433bbf01269bf4f25849
I20260925 15:13:43.522001 20418 ts_tablet_manager.cc:1939] T e3527aa97b18433bbf01269bf4f25849 P bf7cab7961534d01af401cdd0fc4c5ff: Deleting consensus metadata
I20260925 15:13:43.522503 20160 tablet_service.cc:1467] Processing CreateTablet for tablet e06b6369212d4277aa3452c4a11865cb (DEFAULT_TABLE table=test-table [id=c20cbef31ff7414fb90de53e1ae98bc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:13:43.522927 20160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e06b6369212d4277aa3452c4a11865cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:13:43.525858 20393 tablet_bootstrap.cc:492] T e06b6369212d4277aa3452c4a11865cb P c2e6c5753e2546d9bc00393d25672551: Bootstrap starting.
I20260925 15:13:43.526319 19869 catalog_manager.cc:4841] TS bf7cab7961534d01af401cdd0fc4c5ff (127.12.8.2:36763): tablet e3527aa97b18433bbf01269bf4f25849 (table test-table [id=51ed340122e140e48bce487161435930]) successfully deleted
I20260925 15:13:43.527349 20393 tablet_bootstrap.cc:654] T e06b6369212d4277aa3452c4a11865cb P c2e6c5753e2546d9bc00393d25672551: Neither blocks nor log segments found. Creating new log.
I20260925 15:13:43.529215 20395 tablet_bootstrap.cc:492] T e06b6369212d4277aa3452c4a11865cb P b7427e659a2b4fff8dc2e523e45d0718: Bootstrap starting.
I20260925 15:13:43.531324 20395 tablet_bootstrap.cc:654] T e06b6369212d4277aa3452c4a11865cb P b7427e659a2b4fff8dc2e523e45d0718: Neither blocks nor log segments found. Creating new log.
I20260925 15:13:43.533218 20417 tablet_replica.cc:330] T e3527aa97b18433bbf01269bf4f25849 P c2e6c5753e2546d9bc00393d25672551: stopping tablet replica
I20260925 15:13:43.533553 20394 tablet_bootstrap.cc:492] T e06b6369212d4277aa3452c4a11865cb P bf7cab7961534d01af401cdd0fc4c5ff: Bootstrap starting.
I20260925 15:13:43.533668 20417 raft_consensus.cc:2229] T e3527aa97b18433bbf01269bf4f25849 P c2e6c5753e2546d9bc00393d25672551 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:13:43.534173 20417 raft_consensus.cc:2258] T e3527aa97b18433bbf01269bf4f25849 P c2e6c5753e2546d9bc00393d25672551 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:13:43.535462 20394 tablet_bootstrap.cc:654] T e06b6369212d4277aa3452c4a11865cb P bf7cab7961534d01af401cdd0fc4c5ff: Neither blocks nor log segments found. Creating new log.
I20260925 15:13:43.535845 20393 tablet_bootstrap.cc:492] T e06b6369212d4277aa3452c4a11865cb P c2e6c5753e2546d9bc00393d25672551: No bootstrap required, opened a new log
I20260925 15:13:43.535971 20393 ts_tablet_manager.cc:1397] T e06b6369212d4277aa3452c4a11865cb P c2e6c5753e2546d9bc00393d25672551: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260925 15:13:43.536121 20417 ts_tablet_manager.cc:1905] T e3527aa97b18433bbf01269bf4f25849 P c2e6c5753e2546d9bc00393d25672551: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:13:43.536748 20393 raft_consensus.cc:348] T e06b6369212d4277aa3452c4a11865cb P c2e6c5753e2546d9bc00393d25672551 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2e6c5753e2546d9bc00393d25672551" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 39179 } } peers { permanent_uuid: "bf7cab7961534d01af401cdd0fc4c5ff" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 36763 } } peers { permanent_uuid: "b7427e659a2b4fff8dc2e523e45d0718" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43529 } }
I20260925 15:13:43.536968 20393 raft_consensus.cc:374] T e06b6369212d4277aa3452c4a11865cb P c2e6c5753e2546d9bc00393d25672551 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:13:43.537058 20393 raft_consensus.cc:729] T e06b6369212d4277aa3452c4a11865cb P c2e6c5753e2546d9bc00393d25672551 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2e6c5753e2546d9bc00393d25672551, State: Initialized, Role: FOLLOWER
I20260925 15:13:43.537298 20393 consensus_queue.cc:260] T e06b6369212d4277aa3452c4a11865cb P c2e6c5753e2546d9bc00393d25672551 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2e6c5753e2546d9bc00393d25672551" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 39179 } } peers { permanent_uuid: "bf7cab7961534d01af401cdd0fc4c5ff" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 36763 } } peers { permanent_uuid: "b7427e659a2b4fff8dc2e523e45d0718" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43529 } }
I20260925 15:13:43.538781 20393 ts_tablet_manager.cc:1428] T e06b6369212d4277aa3452c4a11865cb P c2e6c5753e2546d9bc00393d25672551: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260925 15:13:43.539930 20395 tablet_bootstrap.cc:492] T e06b6369212d4277aa3452c4a11865cb P b7427e659a2b4fff8dc2e523e45d0718: No bootstrap required, opened a new log
I20260925 15:13:43.540060 20395 ts_tablet_manager.cc:1397] T e06b6369212d4277aa3452c4a11865cb P b7427e659a2b4fff8dc2e523e45d0718: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.003s
I20260925 15:13:43.540937 20395 raft_consensus.cc:348] T e06b6369212d4277aa3452c4a11865cb P b7427e659a2b4fff8dc2e523e45d0718 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2e6c5753e2546d9bc00393d25672551" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 39179 } } peers { permanent_uuid: "bf7cab7961534d01af401cdd0fc4c5ff" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 36763 } } peers { permanent_uuid: "b7427e659a2b4fff8dc2e523e45d0718" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43529 } }
I20260925 15:13:43.541213 20395 raft_consensus.cc:374] T e06b6369212d4277aa3452c4a11865cb P b7427e659a2b4fff8dc2e523e45d0718 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:13:43.541330 20395 raft_consensus.cc:729] T e06b6369212d4277aa3452c4a11865cb P b7427e659a2b4fff8dc2e523e45d0718 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7427e659a2b4fff8dc2e523e45d0718, State: Initialized, Role: FOLLOWER
I20260925 15:13:43.541580 20395 consensus_queue.cc:260] T e06b6369212d4277aa3452c4a11865cb P b7427e659a2b4fff8dc2e523e45d0718 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2e6c5753e2546d9bc00393d25672551" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 39179 } } peers { permanent_uuid: "bf7cab7961534d01af401cdd0fc4c5ff" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 36763 } } peers { permanent_uuid: "b7427e659a2b4fff8dc2e523e45d0718" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43529 } }
I20260925 15:13:43.542277 20395 ts_tablet_manager.cc:1428] T e06b6369212d4277aa3452c4a11865cb P b7427e659a2b4fff8dc2e523e45d0718: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260925 15:13:43.543730 20417 ts_tablet_manager.cc:1918] T e3527aa97b18433bbf01269bf4f25849 P c2e6c5753e2546d9bc00393d25672551: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:13:43.543910 20417 log.cc:1196] T e3527aa97b18433bbf01269bf4f25849 P c2e6c5753e2546d9bc00393d25672551: Deleting WAL directory at /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/minicluster-data/ts-0/wal/wals/e3527aa97b18433bbf01269bf4f25849
I20260925 15:13:43.544533 20417 ts_tablet_manager.cc:1939] T e3527aa97b18433bbf01269bf4f25849 P c2e6c5753e2546d9bc00393d25672551: Deleting consensus metadata
I20260925 15:13:43.546578 19870 catalog_manager.cc:4841] TS c2e6c5753e2546d9bc00393d25672551 (127.12.8.1:39179): tablet e3527aa97b18433bbf01269bf4f25849 (table test-table [id=51ed340122e140e48bce487161435930]) successfully deleted
I20260925 15:13:43.549661 20394 tablet_bootstrap.cc:492] T e06b6369212d4277aa3452c4a11865cb P bf7cab7961534d01af401cdd0fc4c5ff: No bootstrap required, opened a new log
I20260925 15:13:43.549806 20394 ts_tablet_manager.cc:1397] T e06b6369212d4277aa3452c4a11865cb P bf7cab7961534d01af401cdd0fc4c5ff: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.007s
I20260925 15:13:43.550505 20394 raft_consensus.cc:348] T e06b6369212d4277aa3452c4a11865cb P bf7cab7961534d01af401cdd0fc4c5ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2e6c5753e2546d9bc00393d25672551" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 39179 } } peers { permanent_uuid: "bf7cab7961534d01af401cdd0fc4c5ff" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 36763 } } peers { permanent_uuid: "b7427e659a2b4fff8dc2e523e45d0718" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43529 } }
I20260925 15:13:43.550668 20394 raft_consensus.cc:374] T e06b6369212d4277aa3452c4a11865cb P bf7cab7961534d01af401cdd0fc4c5ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:13:43.550808 20394 raft_consensus.cc:729] T e06b6369212d4277aa3452c4a11865cb P bf7cab7961534d01af401cdd0fc4c5ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf7cab7961534d01af401cdd0fc4c5ff, State: Initialized, Role: FOLLOWER
I20260925 15:13:43.551038 20394 consensus_queue.cc:260] T e06b6369212d4277aa3452c4a11865cb P bf7cab7961534d01af401cdd0fc4c5ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2e6c5753e2546d9bc00393d25672551" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 39179 } } peers { permanent_uuid: "bf7cab7961534d01af401cdd0fc4c5ff" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 36763 } } peers { permanent_uuid: "b7427e659a2b4fff8dc2e523e45d0718" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43529 } }
I20260925 15:13:43.551594 20394 ts_tablet_manager.cc:1428] T e06b6369212d4277aa3452c4a11865cb P bf7cab7961534d01af401cdd0fc4c5ff: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260925 15:13:43.553054 20401 raft_consensus.cc:482] T e06b6369212d4277aa3452c4a11865cb P bf7cab7961534d01af401cdd0fc4c5ff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:13:43.553216 20401 raft_consensus.cc:504] T e06b6369212d4277aa3452c4a11865cb P bf7cab7961534d01af401cdd0fc4c5ff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2e6c5753e2546d9bc00393d25672551" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 39179 } } peers { permanent_uuid: "bf7cab7961534d01af401cdd0fc4c5ff" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 36763 } } peers { permanent_uuid: "b7427e659a2b4fff8dc2e523e45d0718" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43529 } }
I20260925 15:13:43.553763 20401 leader_election.cc:288] T e06b6369212d4277aa3452c4a11865cb P bf7cab7961534d01af401cdd0fc4c5ff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2e6c5753e2546d9bc00393d25672551 (127.12.8.1:39179), b7427e659a2b4fff8dc2e523e45d0718 (127.12.8.3:43529)
I20260925 15:13:43.554117 20046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e06b6369212d4277aa3452c4a11865cb" candidate_uuid: "bf7cab7961534d01af401cdd0fc4c5ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2e6c5753e2546d9bc00393d25672551" is_pre_election: true
I20260925 15:13:43.554286 20319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e06b6369212d4277aa3452c4a11865cb" candidate_uuid: "bf7cab7961534d01af401cdd0fc4c5ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7427e659a2b4fff8dc2e523e45d0718" is_pre_election: true
I20260925 15:13:43.554365 20046 raft_consensus.cc:2454] T e06b6369212d4277aa3452c4a11865cb P c2e6c5753e2546d9bc00393d25672551 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf7cab7961534d01af401cdd0fc4c5ff in term 0.
I20260925 15:13:43.554505 20319 raft_consensus.cc:2454] T e06b6369212d4277aa3452c4a11865cb P b7427e659a2b4fff8dc2e523e45d0718 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf7cab7961534d01af401cdd0fc4c5ff in term 0.
I20260925 15:13:43.554795 20113 leader_election.cc:302] T e06b6369212d4277aa3452c4a11865cb P bf7cab7961534d01af401cdd0fc4c5ff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bf7cab7961534d01af401cdd0fc4c5ff, c2e6c5753e2546d9bc00393d25672551; no voters:
I20260925 15:13:43.555015 20401 raft_consensus.cc:2781] T e06b6369212d4277aa3452c4a11865cb P bf7cab7961534d01af401cdd0fc4c5ff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:13:43.555135 20401 raft_consensus.cc:482] T e06b6369212d4277aa3452c4a11865cb P bf7cab7961534d01af401cdd0fc4c5ff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:13:43.555204 20401 raft_consensus.cc:3037] T e06b6369212d4277aa3452c4a11865cb P bf7cab7961534d01af401cdd0fc4c5ff [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:13:43.557377 20401 raft_consensus.cc:504] T e06b6369212d4277aa3452c4a11865cb P bf7cab7961534d01af401cdd0fc4c5ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2e6c5753e2546d9bc00393d25672551" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 39179 } } peers { permanent_uuid: "bf7cab7961534d01af401cdd0fc4c5ff" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 36763 } } peers { permanent_uuid: "b7427e659a2b4fff8dc2e523e45d0718" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43529 } }
I20260925 15:13:43.557941 20401 leader_election.cc:288] T e06b6369212d4277aa3452c4a11865cb P bf7cab7961534d01af401cdd0fc4c5ff [CANDIDATE]: Term 1 election: Requested vote from peers c2e6c5753e2546d9bc00393d25672551 (127.12.8.1:39179), b7427e659a2b4fff8dc2e523e45d0718 (127.12.8.3:43529)
I20260925 15:13:43.558334 20046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e06b6369212d4277aa3452c4a11865cb" candidate_uuid: "bf7cab7961534d01af401cdd0fc4c5ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2e6c5753e2546d9bc00393d25672551"
I20260925 15:13:43.558427 20319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e06b6369212d4277aa3452c4a11865cb" candidate_uuid: "bf7cab7961534d01af401cdd0fc4c5ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7427e659a2b4fff8dc2e523e45d0718"
I20260925 15:13:43.558514 20046 raft_consensus.cc:3037] T e06b6369212d4277aa3452c4a11865cb P c2e6c5753e2546d9bc00393d25672551 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:13:43.558619 20319 raft_consensus.cc:3037] T e06b6369212d4277aa3452c4a11865cb P b7427e659a2b4fff8dc2e523e45d0718 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:13:43.560587 20046 raft_consensus.cc:2454] T e06b6369212d4277aa3452c4a11865cb P c2e6c5753e2546d9bc00393d25672551 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf7cab7961534d01af401cdd0fc4c5ff in term 1.
I20260925 15:13:43.560956 20113 leader_election.cc:302] T e06b6369212d4277aa3452c4a11865cb P bf7cab7961534d01af401cdd0fc4c5ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bf7cab7961534d01af401cdd0fc4c5ff, c2e6c5753e2546d9bc00393d25672551; no voters:
I20260925 15:13:43.561213 20401 raft_consensus.cc:2781] T e06b6369212d4277aa3452c4a11865cb P bf7cab7961534d01af401cdd0fc4c5ff [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:13:43.561391 20401 raft_consensus.cc:686] T e06b6369212d4277aa3452c4a11865cb P bf7cab7961534d01af401cdd0fc4c5ff [term 1 LEADER]: Becoming Leader. State: Replica: bf7cab7961534d01af401cdd0fc4c5ff, State: Running, Role: LEADER
I20260925 15:13:43.561589 20319 raft_consensus.cc:2454] T e06b6369212d4277aa3452c4a11865cb P b7427e659a2b4fff8dc2e523e45d0718 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf7cab7961534d01af401cdd0fc4c5ff in term 1.
I20260925 15:13:43.561882 20401 consensus_queue.cc:237] T e06b6369212d4277aa3452c4a11865cb P bf7cab7961534d01af401cdd0fc4c5ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2e6c5753e2546d9bc00393d25672551" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 39179 } } peers { permanent_uuid: "bf7cab7961534d01af401cdd0fc4c5ff" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 36763 } } peers { permanent_uuid: "b7427e659a2b4fff8dc2e523e45d0718" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43529 } }
I20260925 15:13:43.564217 19883 catalog_manager.cc:5495] T e06b6369212d4277aa3452c4a11865cb P bf7cab7961534d01af401cdd0fc4c5ff reported cstate change: term changed from 0 to 1, leader changed from <none> to bf7cab7961534d01af401cdd0fc4c5ff (127.12.8.2). New cstate: current_term: 1 leader_uuid: "bf7cab7961534d01af401cdd0fc4c5ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2e6c5753e2546d9bc00393d25672551" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 39179 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf7cab7961534d01af401cdd0fc4c5ff" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 36763 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b7427e659a2b4fff8dc2e523e45d0718" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43529 } health_report { overall_health: UNKNOWN } } }
Sep 25 15:13:43 dist-test-slave-p468 krb5kdc[19839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349221, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.8.62@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM
I20260925 15:13:43.607684 19978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:43.592864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59790 (local address 127.12.8.62:32861)
0925 15:13:43.593122 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:43.593131 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:43.593759 (+ 628us) server_negotiation.cc:407] Connection header received
0925 15:13:43.593899 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:43.593905 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:43.593979 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:43.594092 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:43.595258 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:43.596447 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:43.597777 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:43.598471 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:43.598615 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:43.602072 (+ 3457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:43.602079 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:43.602084 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:43.602151 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:43.604970 (+ 2819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:43.605615 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:43.605622 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:43.605626 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:43.605738 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:43.606086 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:43.606092 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:43.606098 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:43.606397 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:43.606539 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:43.606951 (+ 412us) server_negotiation.cc:299] Negotiation successful
0925 15:13:43.607339 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":177,"server-negotiator.queue_time_us":101}
I20260925 15:13:43.607705 20428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:43.592930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:32861 (local address 127.0.0.1:59790)
0925 15:13:43.593658 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:43.593674 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:43.593792 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:43.594112 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:43.594115 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:43.594852 (+ 737us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:13:43.595118 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:43.595127 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:43.596587 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:43.596592 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:43.597648 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:43.597659 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:43.598789 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:43.598800 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:43.598918 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:43.599580 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:13:43.599596 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:13:43.601874 (+ 2278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:13:43.605102 (+ 3228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:13:43.605115 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:13:43.605122 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:13:43.605504 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:13:43.605853 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:13:43.605857 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:13:43.605861 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:13:43.605989 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:13:43.606516 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:13:43.606524 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:13:43.606699 (+ 175us) client_negotiation.cc:769] Sending connection context
0925 15:13:43.606868 (+ 169us) client_negotiation.cc:240] Negotiation successful
0925 15:13:43.607226 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"mutex_wait_us":21,"thread_start_us":439,"threads_started":1}
I20260925 15:13:43.613255 19883 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.12.8.62@KRBTEST.COM'} at 127.0.0.1:59790:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 15:13:43.614383 19883 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:13:43.620783 20021 tablet_service.cc:1467] Processing CreateTablet for tablet b7f051a542204339af98fc77cace9fac (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=77d54c6d8adb4c3e95d040b75a92b954]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 15:13:43.621004 20160 tablet_service.cc:1467] Processing CreateTablet for tablet b7f051a542204339af98fc77cace9fac (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=77d54c6d8adb4c3e95d040b75a92b954]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 15:13:43.621402 20021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7f051a542204339af98fc77cace9fac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:13:43.621340 20299 tablet_service.cc:1467] Processing CreateTablet for tablet b7f051a542204339af98fc77cace9fac (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=77d54c6d8adb4c3e95d040b75a92b954]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 15:13:43.621579 20160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7f051a542204339af98fc77cace9fac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:13:43.621833 20299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7f051a542204339af98fc77cace9fac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:13:43.628703 20395 tablet_bootstrap.cc:492] T b7f051a542204339af98fc77cace9fac P b7427e659a2b4fff8dc2e523e45d0718: Bootstrap starting.
I20260925 15:13:43.628720 20393 tablet_bootstrap.cc:492] T b7f051a542204339af98fc77cace9fac P c2e6c5753e2546d9bc00393d25672551: Bootstrap starting.
I20260925 15:13:43.628747 20394 tablet_bootstrap.cc:492] T b7f051a542204339af98fc77cace9fac P bf7cab7961534d01af401cdd0fc4c5ff: Bootstrap starting.
I20260925 15:13:43.630429 20393 tablet_bootstrap.cc:654] T b7f051a542204339af98fc77cace9fac P c2e6c5753e2546d9bc00393d25672551: Neither blocks nor log segments found. Creating new log.
I20260925 15:13:43.630625 20394 tablet_bootstrap.cc:654] T b7f051a542204339af98fc77cace9fac P bf7cab7961534d01af401cdd0fc4c5ff: Neither blocks nor log segments found. Creating new log.
I20260925 15:13:43.630913 20395 tablet_bootstrap.cc:654] T b7f051a542204339af98fc77cace9fac P b7427e659a2b4fff8dc2e523e45d0718: Neither blocks nor log segments found. Creating new log.
I20260925 15:13:43.632622 20393 tablet_bootstrap.cc:492] T b7f051a542204339af98fc77cace9fac P c2e6c5753e2546d9bc00393d25672551: No bootstrap required, opened a new log
I20260925 15:13:43.632804 20393 ts_tablet_manager.cc:1397] T b7f051a542204339af98fc77cace9fac P c2e6c5753e2546d9bc00393d25672551: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260925 15:13:43.633184 20395 tablet_bootstrap.cc:492] T b7f051a542204339af98fc77cace9fac P b7427e659a2b4fff8dc2e523e45d0718: No bootstrap required, opened a new log
I20260925 15:13:43.633339 20395 ts_tablet_manager.cc:1397] T b7f051a542204339af98fc77cace9fac P b7427e659a2b4fff8dc2e523e45d0718: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20260925 15:13:43.633535 20393 raft_consensus.cc:348] T b7f051a542204339af98fc77cace9fac P c2e6c5753e2546d9bc00393d25672551 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2e6c5753e2546d9bc00393d25672551" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 39179 } } peers { permanent_uuid: "bf7cab7961534d01af401cdd0fc4c5ff" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 36763 } } peers { permanent_uuid: "b7427e659a2b4fff8dc2e523e45d0718" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43529 } }
I20260925 15:13:43.633800 20393 raft_consensus.cc:374] T b7f051a542204339af98fc77cace9fac P c2e6c5753e2546d9bc00393d25672551 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:13:43.633885 20393 raft_consensus.cc:729] T b7f051a542204339af98fc77cace9fac P c2e6c5753e2546d9bc00393d25672551 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2e6c5753e2546d9bc00393d25672551, State: Initialized, Role: FOLLOWER
I20260925 15:13:43.633991 20395 raft_consensus.cc:348] T b7f051a542204339af98fc77cace9fac P b7427e659a2b4fff8dc2e523e45d0718 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2e6c5753e2546d9bc00393d25672551" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 39179 } } peers { permanent_uuid: "bf7cab7961534d01af401cdd0fc4c5ff" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 36763 } } peers { permanent_uuid: "b7427e659a2b4fff8dc2e523e45d0718" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43529 } }
I20260925 15:13:43.634137 20393 consensus_queue.cc:260] T b7f051a542204339af98fc77cace9fac P c2e6c5753e2546d9bc00393d25672551 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2e6c5753e2546d9bc00393d25672551" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 39179 } } peers { permanent_uuid: "bf7cab7961534d01af401cdd0fc4c5ff" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 36763 } } peers { permanent_uuid: "b7427e659a2b4fff8dc2e523e45d0718" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43529 } }
I20260925 15:13:43.634347 20395 raft_consensus.cc:374] T b7f051a542204339af98fc77cace9fac P b7427e659a2b4fff8dc2e523e45d0718 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:13:43.634421 20395 raft_consensus.cc:729] T b7f051a542204339af98fc77cace9fac P b7427e659a2b4fff8dc2e523e45d0718 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7427e659a2b4fff8dc2e523e45d0718, State: Initialized, Role: FOLLOWER
I20260925 15:13:43.634780 20393 ts_tablet_manager.cc:1428] T b7f051a542204339af98fc77cace9fac P c2e6c5753e2546d9bc00393d25672551: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 15:13:43.634688 20395 consensus_queue.cc:260] T b7f051a542204339af98fc77cace9fac P b7427e659a2b4fff8dc2e523e45d0718 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2e6c5753e2546d9bc00393d25672551" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 39179 } } peers { permanent_uuid: "bf7cab7961534d01af401cdd0fc4c5ff" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 36763 } } peers { permanent_uuid: "b7427e659a2b4fff8dc2e523e45d0718" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43529 } }
I20260925 15:13:43.635100 20400 tablet_replica.cc:439] T b7f051a542204339af98fc77cace9fac P c2e6c5753e2546d9bc00393d25672551: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2e6c5753e2546d9bc00393d25672551" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 39179 } } peers { permanent_uuid: "bf7cab7961534d01af401cdd0fc4c5ff" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 36763 } } peers { permanent_uuid: "b7427e659a2b4fff8dc2e523e45d0718" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43529 } } }
I20260925 15:13:43.635106 20399 tablet_replica.cc:439] T b7f051a542204339af98fc77cace9fac P b7427e659a2b4fff8dc2e523e45d0718: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2e6c5753e2546d9bc00393d25672551" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 39179 } } peers { permanent_uuid: "bf7cab7961534d01af401cdd0fc4c5ff" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 36763 } } peers { permanent_uuid: "b7427e659a2b4fff8dc2e523e45d0718" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43529 } } }
I20260925 15:13:43.635444 20399 tablet_replica.cc:442] T b7f051a542204339af98fc77cace9fac P b7427e659a2b4fff8dc2e523e45d0718: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:13:43.635447 20400 tablet_replica.cc:442] T b7f051a542204339af98fc77cace9fac P c2e6c5753e2546d9bc00393d25672551: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:13:43.635547 20394 tablet_bootstrap.cc:492] T b7f051a542204339af98fc77cace9fac P bf7cab7961534d01af401cdd0fc4c5ff: No bootstrap required, opened a new log
I20260925 15:13:43.635753 20394 ts_tablet_manager.cc:1397] T b7f051a542204339af98fc77cace9fac P bf7cab7961534d01af401cdd0fc4c5ff: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260925 15:13:43.635844 20395 ts_tablet_manager.cc:1428] T b7f051a542204339af98fc77cace9fac P b7427e659a2b4fff8dc2e523e45d0718: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 15:13:43.636883 20394 raft_consensus.cc:348] T b7f051a542204339af98fc77cace9fac P bf7cab7961534d01af401cdd0fc4c5ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2e6c5753e2546d9bc00393d25672551" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 39179 } } peers { permanent_uuid: "bf7cab7961534d01af401cdd0fc4c5ff" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 36763 } } peers { permanent_uuid: "b7427e659a2b4fff8dc2e523e45d0718" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43529 } }
I20260925 15:13:43.637208 20394 raft_consensus.cc:374] T b7f051a542204339af98fc77cace9fac P bf7cab7961534d01af401cdd0fc4c5ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:13:43.637349 20394 raft_consensus.cc:729] T b7f051a542204339af98fc77cace9fac P bf7cab7961534d01af401cdd0fc4c5ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf7cab7961534d01af401cdd0fc4c5ff, State: Initialized, Role: FOLLOWER
I20260925 15:13:43.637681 20394 consensus_queue.cc:260] T b7f051a542204339af98fc77cace9fac P bf7cab7961534d01af401cdd0fc4c5ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2e6c5753e2546d9bc00393d25672551" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 39179 } } peers { permanent_uuid: "bf7cab7961534d01af401cdd0fc4c5ff" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 36763 } } peers { permanent_uuid: "b7427e659a2b4fff8dc2e523e45d0718" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43529 } }
I20260925 15:13:43.638371 20394 ts_tablet_manager.cc:1428] T b7f051a542204339af98fc77cace9fac P bf7cab7961534d01af401cdd0fc4c5ff: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 15:13:43.638229 20401 tablet_replica.cc:439] T b7f051a542204339af98fc77cace9fac P bf7cab7961534d01af401cdd0fc4c5ff: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2e6c5753e2546d9bc00393d25672551" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 39179 } } peers { permanent_uuid: "bf7cab7961534d01af401cdd0fc4c5ff" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 36763 } } peers { permanent_uuid: "b7427e659a2b4fff8dc2e523e45d0718" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43529 } } }
I20260925 15:13:43.638645 20401 tablet_replica.cc:442] T b7f051a542204339af98fc77cace9fac P bf7cab7961534d01af401cdd0fc4c5ff: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:13:43.722859 20401 raft_consensus.cc:482] T b7f051a542204339af98fc77cace9fac P bf7cab7961534d01af401cdd0fc4c5ff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:13:43.723112 20401 raft_consensus.cc:504] T b7f051a542204339af98fc77cace9fac P bf7cab7961534d01af401cdd0fc4c5ff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2e6c5753e2546d9bc00393d25672551" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 39179 } } peers { permanent_uuid: "bf7cab7961534d01af401cdd0fc4c5ff" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 36763 } } peers { permanent_uuid: "b7427e659a2b4fff8dc2e523e45d0718" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43529 } }
I20260925 15:13:43.723812 20401 leader_election.cc:288] T b7f051a542204339af98fc77cace9fac P bf7cab7961534d01af401cdd0fc4c5ff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2e6c5753e2546d9bc00393d25672551 (127.12.8.1:39179), b7427e659a2b4fff8dc2e523e45d0718 (127.12.8.3:43529)
I20260925 15:13:43.724361 20046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7f051a542204339af98fc77cace9fac" candidate_uuid: "bf7cab7961534d01af401cdd0fc4c5ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2e6c5753e2546d9bc00393d25672551" is_pre_election: true
I20260925 15:13:43.724411 20319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7f051a542204339af98fc77cace9fac" candidate_uuid: "bf7cab7961534d01af401cdd0fc4c5ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7427e659a2b4fff8dc2e523e45d0718" is_pre_election: true
I20260925 15:13:43.724638 20319 raft_consensus.cc:2454] T b7f051a542204339af98fc77cace9fac P b7427e659a2b4fff8dc2e523e45d0718 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf7cab7961534d01af401cdd0fc4c5ff in term 0.
I20260925 15:13:43.724658 20046 raft_consensus.cc:2454] T b7f051a542204339af98fc77cace9fac P c2e6c5753e2546d9bc00393d25672551 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf7cab7961534d01af401cdd0fc4c5ff in term 0.
I20260925 15:13:43.725076 20112 leader_election.cc:302] T b7f051a542204339af98fc77cace9fac P bf7cab7961534d01af401cdd0fc4c5ff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7427e659a2b4fff8dc2e523e45d0718, bf7cab7961534d01af401cdd0fc4c5ff; no voters:
I20260925 15:13:43.725404 20401 raft_consensus.cc:2781] T b7f051a542204339af98fc77cace9fac P bf7cab7961534d01af401cdd0fc4c5ff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:13:43.725518 20401 raft_consensus.cc:482] T b7f051a542204339af98fc77cace9fac P bf7cab7961534d01af401cdd0fc4c5ff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:13:43.725663 20401 raft_consensus.cc:3037] T b7f051a542204339af98fc77cace9fac P bf7cab7961534d01af401cdd0fc4c5ff [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:13:43.727890 20401 raft_consensus.cc:504] T b7f051a542204339af98fc77cace9fac P bf7cab7961534d01af401cdd0fc4c5ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2e6c5753e2546d9bc00393d25672551" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 39179 } } peers { permanent_uuid: "bf7cab7961534d01af401cdd0fc4c5ff" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 36763 } } peers { permanent_uuid: "b7427e659a2b4fff8dc2e523e45d0718" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43529 } }
I20260925 15:13:43.728417 20401 leader_election.cc:288] T b7f051a542204339af98fc77cace9fac P bf7cab7961534d01af401cdd0fc4c5ff [CANDIDATE]: Term 1 election: Requested vote from peers c2e6c5753e2546d9bc00393d25672551 (127.12.8.1:39179), b7427e659a2b4fff8dc2e523e45d0718 (127.12.8.3:43529)
I20260925 15:13:43.728823 20046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7f051a542204339af98fc77cace9fac" candidate_uuid: "bf7cab7961534d01af401cdd0fc4c5ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2e6c5753e2546d9bc00393d25672551"
I20260925 15:13:43.728850 20319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7f051a542204339af98fc77cace9fac" candidate_uuid: "bf7cab7961534d01af401cdd0fc4c5ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7427e659a2b4fff8dc2e523e45d0718"
I20260925 15:13:43.729056 20046 raft_consensus.cc:3037] T b7f051a542204339af98fc77cace9fac P c2e6c5753e2546d9bc00393d25672551 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:13:43.729069 20319 raft_consensus.cc:3037] T b7f051a542204339af98fc77cace9fac P b7427e659a2b4fff8dc2e523e45d0718 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:13:43.731932 20319 raft_consensus.cc:2454] T b7f051a542204339af98fc77cace9fac P b7427e659a2b4fff8dc2e523e45d0718 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf7cab7961534d01af401cdd0fc4c5ff in term 1.
I20260925 15:13:43.731932 20046 raft_consensus.cc:2454] T b7f051a542204339af98fc77cace9fac P c2e6c5753e2546d9bc00393d25672551 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf7cab7961534d01af401cdd0fc4c5ff in term 1.
I20260925 15:13:43.732323 20112 leader_election.cc:302] T b7f051a542204339af98fc77cace9fac P bf7cab7961534d01af401cdd0fc4c5ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7427e659a2b4fff8dc2e523e45d0718, bf7cab7961534d01af401cdd0fc4c5ff; no voters:
I20260925 15:13:43.732580 20401 raft_consensus.cc:2781] T b7f051a542204339af98fc77cace9fac P bf7cab7961534d01af401cdd0fc4c5ff [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:13:43.732784 20401 raft_consensus.cc:686] T b7f051a542204339af98fc77cace9fac P bf7cab7961534d01af401cdd0fc4c5ff [term 1 LEADER]: Becoming Leader. State: Replica: bf7cab7961534d01af401cdd0fc4c5ff, State: Running, Role: LEADER
I20260925 15:13:43.733070 20401 consensus_queue.cc:237] T b7f051a542204339af98fc77cace9fac P bf7cab7961534d01af401cdd0fc4c5ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2e6c5753e2546d9bc00393d25672551" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 39179 } } peers { permanent_uuid: "bf7cab7961534d01af401cdd0fc4c5ff" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 36763 } } peers { permanent_uuid: "b7427e659a2b4fff8dc2e523e45d0718" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43529 } }
I20260925 15:13:43.734779 20404 tablet_replica.cc:439] T b7f051a542204339af98fc77cace9fac P bf7cab7961534d01af401cdd0fc4c5ff: TxnStatusTablet state changed. Reason: New leader bf7cab7961534d01af401cdd0fc4c5ff. Latest consensus state: current_term: 1 leader_uuid: "bf7cab7961534d01af401cdd0fc4c5ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2e6c5753e2546d9bc00393d25672551" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 39179 } } peers { permanent_uuid: "bf7cab7961534d01af401cdd0fc4c5ff" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 36763 } } peers { permanent_uuid: "b7427e659a2b4fff8dc2e523e45d0718" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43529 } } }
I20260925 15:13:43.735128 20404 tablet_replica.cc:442] T b7f051a542204339af98fc77cace9fac P bf7cab7961534d01af401cdd0fc4c5ff: This TxnStatusTablet replica's current role is: LEADER
I20260925 15:13:43.736130 19882 catalog_manager.cc:5495] T b7f051a542204339af98fc77cace9fac P bf7cab7961534d01af401cdd0fc4c5ff reported cstate change: term changed from 0 to 1, leader changed from <none> to bf7cab7961534d01af401cdd0fc4c5ff (127.12.8.2). New cstate: current_term: 1 leader_uuid: "bf7cab7961534d01af401cdd0fc4c5ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2e6c5753e2546d9bc00393d25672551" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 39179 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf7cab7961534d01af401cdd0fc4c5ff" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 36763 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b7427e659a2b4fff8dc2e523e45d0718" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43529 } health_report { overall_health: UNKNOWN } } }
I20260925 15:13:43.738754 20433 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 15:13:43.739012 20433 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 15:13:43.767148 20392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:43.762066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35700 (local address 127.12.8.2:36763)
0925 15:13:43.762256 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:43.762262 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:43.762425 (+ 163us) server_negotiation.cc:407] Connection header received
0925 15:13:43.762682 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:43.762687 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:43.762757 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:43.762874 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:13:43.763230 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:43.764043 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:43.765329 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:43.765769 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:43.765925 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:43.766390 (+ 465us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:13:43.766511 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:13:43.766606 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:43.766729 (+ 123us) server_negotiation.cc:299] Negotiation successful
0925 15:13:43.766801 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 15:13:43.767145 20428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:43.762080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.2:36763 (local address 127.0.0.1:35700)
0925 15:13:43.762334 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:43.762350 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:43.762514 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:43.762892 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:43.762897 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:43.762908 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 15:13:43.763101 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:43.763109 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:43.764183 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:43.764188 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:43.765170 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:43.765181 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:43.765932 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:43.765938 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:43.766117 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:43.766170 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 15:13:43.766607 (+ 437us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 15:13:43.766628 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 15:13:43.766706 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 15:13:43.766848 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260925 15:13:43.770830 20319 raft_consensus.cc:1261] T b7f051a542204339af98fc77cace9fac P b7427e659a2b4fff8dc2e523e45d0718 [term 1 FOLLOWER]: Refusing update from remote peer bf7cab7961534d01af401cdd0fc4c5ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:13:43.770830 20046 raft_consensus.cc:1261] T b7f051a542204339af98fc77cace9fac P c2e6c5753e2546d9bc00393d25672551 [term 1 FOLLOWER]: Refusing update from remote peer bf7cab7961534d01af401cdd0fc4c5ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:13:43.771502 20401 consensus_queue.cc:1035] T b7f051a542204339af98fc77cace9fac P bf7cab7961534d01af401cdd0fc4c5ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2e6c5753e2546d9bc00393d25672551" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 39179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:13:43.771765 20404 consensus_queue.cc:1035] T b7f051a542204339af98fc77cace9fac P bf7cab7961534d01af401cdd0fc4c5ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7427e659a2b4fff8dc2e523e45d0718" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:13:43.776024 20400 tablet_replica.cc:439] T b7f051a542204339af98fc77cace9fac P c2e6c5753e2546d9bc00393d25672551: TxnStatusTablet state changed. Reason: New leader bf7cab7961534d01af401cdd0fc4c5ff. Latest consensus state: current_term: 1 leader_uuid: "bf7cab7961534d01af401cdd0fc4c5ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2e6c5753e2546d9bc00393d25672551" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 39179 } } peers { permanent_uuid: "bf7cab7961534d01af401cdd0fc4c5ff" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 36763 } } peers { permanent_uuid: "b7427e659a2b4fff8dc2e523e45d0718" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43529 } } }
I20260925 15:13:43.776307 20400 tablet_replica.cc:442] T b7f051a542204339af98fc77cace9fac P c2e6c5753e2546d9bc00393d25672551: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:13:43.777751 20401 tablet_replica.cc:439] T b7f051a542204339af98fc77cace9fac P bf7cab7961534d01af401cdd0fc4c5ff: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bf7cab7961534d01af401cdd0fc4c5ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2e6c5753e2546d9bc00393d25672551" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 39179 } } peers { permanent_uuid: "bf7cab7961534d01af401cdd0fc4c5ff" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 36763 } } peers { permanent_uuid: "b7427e659a2b4fff8dc2e523e45d0718" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43529 } } }
I20260925 15:13:43.777905 20407 mvcc.cc:205] Tried to move back new op lower bound from 7333270420559466496 to 7333270420413120512. Current Snapshot: MvccSnapshot[applied={T|T < 7333270420559466496}]
I20260925 15:13:43.778072 20401 tablet_replica.cc:442] T b7f051a542204339af98fc77cace9fac P bf7cab7961534d01af401cdd0fc4c5ff: This TxnStatusTablet replica's current role is: LEADER
I20260925 15:13:43.779549 20400 tablet_replica.cc:439] T b7f051a542204339af98fc77cace9fac P c2e6c5753e2546d9bc00393d25672551: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bf7cab7961534d01af401cdd0fc4c5ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2e6c5753e2546d9bc00393d25672551" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 39179 } } peers { permanent_uuid: "bf7cab7961534d01af401cdd0fc4c5ff" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 36763 } } peers { permanent_uuid: "b7427e659a2b4fff8dc2e523e45d0718" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43529 } } }
I20260925 15:13:43.779762 20400 tablet_replica.cc:442] T b7f051a542204339af98fc77cace9fac P c2e6c5753e2546d9bc00393d25672551: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:13:43.784540 20399 tablet_replica.cc:439] T b7f051a542204339af98fc77cace9fac P b7427e659a2b4fff8dc2e523e45d0718: TxnStatusTablet state changed. Reason: New leader bf7cab7961534d01af401cdd0fc4c5ff. Latest consensus state: current_term: 1 leader_uuid: "bf7cab7961534d01af401cdd0fc4c5ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2e6c5753e2546d9bc00393d25672551" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 39179 } } peers { permanent_uuid: "bf7cab7961534d01af401cdd0fc4c5ff" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 36763 } } peers { permanent_uuid: "b7427e659a2b4fff8dc2e523e45d0718" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43529 } } }
I20260925 15:13:43.784813 20399 tablet_replica.cc:442] T b7f051a542204339af98fc77cace9fac P b7427e659a2b4fff8dc2e523e45d0718: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:13:43.787628 20401 tablet_replica.cc:439] T b7f051a542204339af98fc77cace9fac P bf7cab7961534d01af401cdd0fc4c5ff: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bf7cab7961534d01af401cdd0fc4c5ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2e6c5753e2546d9bc00393d25672551" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 39179 } } peers { permanent_uuid: "bf7cab7961534d01af401cdd0fc4c5ff" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 36763 } } peers { permanent_uuid: "b7427e659a2b4fff8dc2e523e45d0718" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43529 } } }
I20260925 15:13:43.787923 20401 tablet_replica.cc:442] T b7f051a542204339af98fc77cace9fac P bf7cab7961534d01af401cdd0fc4c5ff: This TxnStatusTablet replica's current role is: LEADER
I20260925 15:13:43.792299 20399 tablet_replica.cc:439] T b7f051a542204339af98fc77cace9fac P b7427e659a2b4fff8dc2e523e45d0718: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bf7cab7961534d01af401cdd0fc4c5ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2e6c5753e2546d9bc00393d25672551" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 39179 } } peers { permanent_uuid: "bf7cab7961534d01af401cdd0fc4c5ff" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 36763 } } peers { permanent_uuid: "b7427e659a2b4fff8dc2e523e45d0718" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43529 } } }
I20260925 15:13:43.792585 20399 tablet_replica.cc:442] T b7f051a542204339af98fc77cace9fac P b7427e659a2b4fff8dc2e523e45d0718: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:13:43.809206 20392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:43.803120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.2:53199 (local address 127.12.8.2:36763)
0925 15:13:43.803310 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:43.803316 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:43.804120 (+ 804us) server_negotiation.cc:407] Connection header received
0925 15:13:43.804323 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:43.804328 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:43.804399 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:43.804496 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:13:43.804990 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:43.805726 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:43.807341 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:43.807786 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:43.807933 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:43.808368 (+ 435us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:13:43.808484 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:13:43.808572 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:43.808708 (+ 136us) server_negotiation.cc:299] Negotiation successful
0925 15:13:43.808832 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":51}
I20260925 15:13:43.809542 20440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:43.803195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.2:36763 (local address 127.12.8.2:53199)
0925 15:13:43.804029 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:43.804044 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:43.804199 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:43.804506 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:43.804510 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:43.804520 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 15:13:43.804854 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:43.804865 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:43.805867 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:43.805873 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:43.807182 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:43.807194 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:43.807976 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:43.807982 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:43.808129 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:43.808178 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 15:13:43.808571 (+ 393us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 15:13:43.808589 (+ 18us) client_negotiation.cc:769] Sending connection context
0925 15:13:43.808687 (+ 98us) client_negotiation.cc:240] Negotiation successful
0925 15:13:43.808867 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":688,"mutex_wait_us":24,"thread_start_us":428,"threads_started":1}
I20260925 15:13:43.818136 20319 raft_consensus.cc:1261] T e06b6369212d4277aa3452c4a11865cb P b7427e659a2b4fff8dc2e523e45d0718 [term 1 FOLLOWER]: Refusing update from remote peer bf7cab7961534d01af401cdd0fc4c5ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:13:43.818442 20046 raft_consensus.cc:1261] T e06b6369212d4277aa3452c4a11865cb P c2e6c5753e2546d9bc00393d25672551 [term 1 FOLLOWER]: Refusing update from remote peer bf7cab7961534d01af401cdd0fc4c5ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:13:43.818635 20401 consensus_queue.cc:1035] T e06b6369212d4277aa3452c4a11865cb P bf7cab7961534d01af401cdd0fc4c5ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7427e659a2b4fff8dc2e523e45d0718" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:13:43.819026 20437 consensus_queue.cc:1035] T e06b6369212d4277aa3452c4a11865cb P bf7cab7961534d01af401cdd0fc4c5ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2e6c5753e2546d9bc00393d25672551" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 39179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:13:43.873063 19882 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58216:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:13:43.873330 19882 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58216:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:13:43.876544 19882 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3312e9eb760f4482923279ed63ae02a8: Sending DeleteTablet for 3 replicas of tablet e06b6369212d4277aa3452c4a11865cb
I20260925 15:13:43.877287 20021 tablet_service.cc:1514] Processing DeleteTablet for tablet e06b6369212d4277aa3452c4a11865cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:13:43 UTC) from {username='oryx', principal='oryx/127.12.8.62@KRBTEST.COM'} at 127.0.0.1:56062
I20260925 15:13:43.877554 20417 tablet_replica.cc:330] T e06b6369212d4277aa3452c4a11865cb P c2e6c5753e2546d9bc00393d25672551: stopping tablet replica
I20260925 15:13:43.877743 20299 tablet_service.cc:1514] Processing DeleteTablet for tablet e06b6369212d4277aa3452c4a11865cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:13:43 UTC) from {username='oryx', principal='oryx/127.12.8.62@KRBTEST.COM'} at 127.0.0.1:56158
I20260925 15:13:43.877771 20417 raft_consensus.cc:2229] T e06b6369212d4277aa3452c4a11865cb P c2e6c5753e2546d9bc00393d25672551 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:13:43.877961 20417 raft_consensus.cc:2258] T e06b6369212d4277aa3452c4a11865cb P c2e6c5753e2546d9bc00393d25672551 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:13:43.877996 20416 tablet_replica.cc:330] T e06b6369212d4277aa3452c4a11865cb P b7427e659a2b4fff8dc2e523e45d0718: stopping tablet replica
I20260925 15:13:43.878178 20416 raft_consensus.cc:2229] T e06b6369212d4277aa3452c4a11865cb P b7427e659a2b4fff8dc2e523e45d0718 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:13:43.878409 20416 raft_consensus.cc:2258] T e06b6369212d4277aa3452c4a11865cb P b7427e659a2b4fff8dc2e523e45d0718 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:13:43.879065 20417 ts_tablet_manager.cc:1905] T e06b6369212d4277aa3452c4a11865cb P c2e6c5753e2546d9bc00393d25672551: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:13:43.879702 20416 ts_tablet_manager.cc:1905] T e06b6369212d4277aa3452c4a11865cb P b7427e659a2b4fff8dc2e523e45d0718: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:13:43.882385 20160 tablet_service.cc:1514] Processing DeleteTablet for tablet e06b6369212d4277aa3452c4a11865cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:13:43 UTC) from {username='oryx', principal='oryx/127.12.8.62@KRBTEST.COM'} at 127.0.0.1:33624
I20260925 15:13:43.882647 20418 tablet_replica.cc:330] T e06b6369212d4277aa3452c4a11865cb P bf7cab7961534d01af401cdd0fc4c5ff: stopping tablet replica
I20260925 15:13:43.882827 20418 raft_consensus.cc:2229] T e06b6369212d4277aa3452c4a11865cb P bf7cab7961534d01af401cdd0fc4c5ff [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:13:43.883131 20418 raft_consensus.cc:2258] T e06b6369212d4277aa3452c4a11865cb P bf7cab7961534d01af401cdd0fc4c5ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:13:43.884387 20418 ts_tablet_manager.cc:1905] T e06b6369212d4277aa3452c4a11865cb P bf7cab7961534d01af401cdd0fc4c5ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:13:43.886152 20417 ts_tablet_manager.cc:1918] T e06b6369212d4277aa3452c4a11865cb P c2e6c5753e2546d9bc00393d25672551: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 15:13:43.886333 20417 log.cc:1196] T e06b6369212d4277aa3452c4a11865cb P c2e6c5753e2546d9bc00393d25672551: Deleting WAL directory at /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/minicluster-data/ts-0/wal/wals/e06b6369212d4277aa3452c4a11865cb
I20260925 15:13:43.886569 20416 ts_tablet_manager.cc:1918] T e06b6369212d4277aa3452c4a11865cb P b7427e659a2b4fff8dc2e523e45d0718: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 15:13:43.886696 20416 log.cc:1196] T e06b6369212d4277aa3452c4a11865cb P b7427e659a2b4fff8dc2e523e45d0718: Deleting WAL directory at /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/minicluster-data/ts-2/wal/wals/e06b6369212d4277aa3452c4a11865cb
I20260925 15:13:43.886762 20417 ts_tablet_manager.cc:1939] T e06b6369212d4277aa3452c4a11865cb P c2e6c5753e2546d9bc00393d25672551: Deleting consensus metadata
I20260925 15:13:43.887115 20416 ts_tablet_manager.cc:1939] T e06b6369212d4277aa3452c4a11865cb P b7427e659a2b4fff8dc2e523e45d0718: Deleting consensus metadata
I20260925 15:13:43.887832 19870 catalog_manager.cc:4841] TS c2e6c5753e2546d9bc00393d25672551 (127.12.8.1:39179): tablet e06b6369212d4277aa3452c4a11865cb (table test-table [id=c20cbef31ff7414fb90de53e1ae98bc1]) successfully deleted
I20260925 15:13:43.888408 19869 catalog_manager.cc:4841] TS b7427e659a2b4fff8dc2e523e45d0718 (127.12.8.3:43529): tablet e06b6369212d4277aa3452c4a11865cb (table test-table [id=c20cbef31ff7414fb90de53e1ae98bc1]) successfully deleted
I20260925 15:13:43.891238 20418 ts_tablet_manager.cc:1918] T e06b6369212d4277aa3452c4a11865cb P bf7cab7961534d01af401cdd0fc4c5ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 15:13:43.891438 20418 log.cc:1196] T e06b6369212d4277aa3452c4a11865cb P bf7cab7961534d01af401cdd0fc4c5ff: Deleting WAL directory at /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/minicluster-data/ts-1/wal/wals/e06b6369212d4277aa3452c4a11865cb
I20260925 15:13:43.891881 20418 ts_tablet_manager.cc:1939] T e06b6369212d4277aa3452c4a11865cb P bf7cab7961534d01af401cdd0fc4c5ff: Deleting consensus metadata
I20260925 15:13:43.892977 19869 catalog_manager.cc:4841] TS bf7cab7961534d01af401cdd0fc4c5ff (127.12.8.2:36763): tablet e06b6369212d4277aa3452c4a11865cb (table test-table [id=c20cbef31ff7414fb90de53e1ae98bc1]) successfully deleted
Sep 25 15:13:43 dist-test-slave-p468 krb5kdc[19839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 15:13:43 dist-test-slave-p468 krb5kdc[19839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:13:43 dist-test-slave-p468 krb5kdc[19839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 15:13:43.946290 19978 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:13:43.934568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59804 (local address 127.12.8.62:32861)
0925 15:13:43.934838 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:43.934844 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:43.935191 (+ 347us) server_negotiation.cc:407] Connection header received
0925 15:13:43.935397 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:43.935402 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:43.935463 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:43.935552 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:43.937273 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:43.938004 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:43.939885 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:43.940376 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:43.940474 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:43.946008 (+ 5534us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59804: BlockingRecv error: recv got EOF from 127.0.0.1:59804 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 25 15:13:43 dist-test-slave-p468 krb5kdc[19839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM
I20260925 15:13:43.965212 19978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:43.949395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59816 (local address 127.12.8.62:32861)
0925 15:13:43.949619 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:43.949624 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:43.949930 (+ 306us) server_negotiation.cc:407] Connection header received
0925 15:13:43.950083 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:43.950087 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:43.950142 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:43.950299 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:43.951677 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:43.952346 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:43.954158 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:43.954669 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:43.954769 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:43.959194 (+ 4425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:43.959201 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:43.959206 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:43.959248 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:43.962557 (+ 3309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:43.963265 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:43.963273 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:43.963277 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:43.963419 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:43.963764 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:43.963768 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:43.963771 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:43.964138 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:43.964374 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:43.964615 (+ 241us) server_negotiation.cc:299] Negotiation successful
0925 15:13:43.964832 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 15:13:43.968742 19882 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59816:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:13:43.969388 19882 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:13:43.976353 20021 tablet_service.cc:1467] Processing CreateTablet for tablet 1730de82547a4ccca398090e71d75807 (DEFAULT_TABLE table=test-table [id=9cadef999c5a4e5498427a8147abb730]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:13:43.976792 20021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1730de82547a4ccca398090e71d75807. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:13:43.976737 20160 tablet_service.cc:1467] Processing CreateTablet for tablet 1730de82547a4ccca398090e71d75807 (DEFAULT_TABLE table=test-table [id=9cadef999c5a4e5498427a8147abb730]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:13:43.976742 20299 tablet_service.cc:1467] Processing CreateTablet for tablet 1730de82547a4ccca398090e71d75807 (DEFAULT_TABLE table=test-table [id=9cadef999c5a4e5498427a8147abb730]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:13:43.977138 20299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1730de82547a4ccca398090e71d75807. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:13:43.977149 20160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1730de82547a4ccca398090e71d75807. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:13:43.983461 20393 tablet_bootstrap.cc:492] T 1730de82547a4ccca398090e71d75807 P c2e6c5753e2546d9bc00393d25672551: Bootstrap starting.
I20260925 15:13:43.984058 20394 tablet_bootstrap.cc:492] T 1730de82547a4ccca398090e71d75807 P bf7cab7961534d01af401cdd0fc4c5ff: Bootstrap starting.
I20260925 15:13:43.985016 20393 tablet_bootstrap.cc:654] T 1730de82547a4ccca398090e71d75807 P c2e6c5753e2546d9bc00393d25672551: Neither blocks nor log segments found. Creating new log.
I20260925 15:13:43.985700 20394 tablet_bootstrap.cc:654] T 1730de82547a4ccca398090e71d75807 P bf7cab7961534d01af401cdd0fc4c5ff: Neither blocks nor log segments found. Creating new log.
I20260925 15:13:43.985989 20395 tablet_bootstrap.cc:492] T 1730de82547a4ccca398090e71d75807 P b7427e659a2b4fff8dc2e523e45d0718: Bootstrap starting.
I20260925 15:13:43.987432 20395 tablet_bootstrap.cc:654] T 1730de82547a4ccca398090e71d75807 P b7427e659a2b4fff8dc2e523e45d0718: Neither blocks nor log segments found. Creating new log.
I20260925 15:13:43.990527 20394 tablet_bootstrap.cc:492] T 1730de82547a4ccca398090e71d75807 P bf7cab7961534d01af401cdd0fc4c5ff: No bootstrap required, opened a new log
I20260925 15:13:43.990751 20394 ts_tablet_manager.cc:1397] T 1730de82547a4ccca398090e71d75807 P bf7cab7961534d01af401cdd0fc4c5ff: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260925 15:13:43.991684 20394 raft_consensus.cc:348] T 1730de82547a4ccca398090e71d75807 P bf7cab7961534d01af401cdd0fc4c5ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2e6c5753e2546d9bc00393d25672551" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 39179 } } peers { permanent_uuid: "bf7cab7961534d01af401cdd0fc4c5ff" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 36763 } } peers { permanent_uuid: "b7427e659a2b4fff8dc2e523e45d0718" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43529 } }
I20260925 15:13:43.992111 20394 raft_consensus.cc:374] T 1730de82547a4ccca398090e71d75807 P bf7cab7961534d01af401cdd0fc4c5ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:13:43.992265 20394 raft_consensus.cc:729] T 1730de82547a4ccca398090e71d75807 P bf7cab7961534d01af401cdd0fc4c5ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf7cab7961534d01af401cdd0fc4c5ff, State: Initialized, Role: FOLLOWER
I20260925 15:13:43.992676 20394 consensus_queue.cc:260] T 1730de82547a4ccca398090e71d75807 P bf7cab7961534d01af401cdd0fc4c5ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2e6c5753e2546d9bc00393d25672551" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 39179 } } peers { permanent_uuid: "bf7cab7961534d01af401cdd0fc4c5ff" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 36763 } } peers { permanent_uuid: "b7427e659a2b4fff8dc2e523e45d0718" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43529 } }
I20260925 15:13:43.996248 20394 ts_tablet_manager.cc:1428] T 1730de82547a4ccca398090e71d75807 P bf7cab7961534d01af401cdd0fc4c5ff: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260925 15:13:44.003700 20395 tablet_bootstrap.cc:492] T 1730de82547a4ccca398090e71d75807 P b7427e659a2b4fff8dc2e523e45d0718: No bootstrap required, opened a new log
I20260925 15:13:44.003876 20395 ts_tablet_manager.cc:1397] T 1730de82547a4ccca398090e71d75807 P b7427e659a2b4fff8dc2e523e45d0718: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.005s
I20260925 15:13:44.004726 20395 raft_consensus.cc:348] T 1730de82547a4ccca398090e71d75807 P b7427e659a2b4fff8dc2e523e45d0718 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2e6c5753e2546d9bc00393d25672551" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 39179 } } peers { permanent_uuid: "bf7cab7961534d01af401cdd0fc4c5ff" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 36763 } } peers { permanent_uuid: "b7427e659a2b4fff8dc2e523e45d0718" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43529 } }
I20260925 15:13:44.004985 20395 raft_consensus.cc:374] T 1730de82547a4ccca398090e71d75807 P b7427e659a2b4fff8dc2e523e45d0718 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:13:44.005086 20395 raft_consensus.cc:729] T 1730de82547a4ccca398090e71d75807 P b7427e659a2b4fff8dc2e523e45d0718 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7427e659a2b4fff8dc2e523e45d0718, State: Initialized, Role: FOLLOWER
I20260925 15:13:44.005370 20395 consensus_queue.cc:260] T 1730de82547a4ccca398090e71d75807 P b7427e659a2b4fff8dc2e523e45d0718 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2e6c5753e2546d9bc00393d25672551" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 39179 } } peers { permanent_uuid: "bf7cab7961534d01af401cdd0fc4c5ff" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 36763 } } peers { permanent_uuid: "b7427e659a2b4fff8dc2e523e45d0718" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43529 } }
I20260925 15:13:44.005868 20393 tablet_bootstrap.cc:492] T 1730de82547a4ccca398090e71d75807 P c2e6c5753e2546d9bc00393d25672551: No bootstrap required, opened a new log
I20260925 15:13:44.006021 20393 ts_tablet_manager.cc:1397] T 1730de82547a4ccca398090e71d75807 P c2e6c5753e2546d9bc00393d25672551: Time spent bootstrapping tablet: real 0.023s user 0.010s sys 0.001s
I20260925 15:13:44.006038 20395 ts_tablet_manager.cc:1428] T 1730de82547a4ccca398090e71d75807 P b7427e659a2b4fff8dc2e523e45d0718: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260925 15:13:44.006798 20393 raft_consensus.cc:348] T 1730de82547a4ccca398090e71d75807 P c2e6c5753e2546d9bc00393d25672551 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2e6c5753e2546d9bc00393d25672551" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 39179 } } peers { permanent_uuid: "bf7cab7961534d01af401cdd0fc4c5ff" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 36763 } } peers { permanent_uuid: "b7427e659a2b4fff8dc2e523e45d0718" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43529 } }
I20260925 15:13:44.007066 20393 raft_consensus.cc:374] T 1730de82547a4ccca398090e71d75807 P c2e6c5753e2546d9bc00393d25672551 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:13:44.007164 20393 raft_consensus.cc:729] T 1730de82547a4ccca398090e71d75807 P c2e6c5753e2546d9bc00393d25672551 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2e6c5753e2546d9bc00393d25672551, State: Initialized, Role: FOLLOWER
I20260925 15:13:44.007380 20393 consensus_queue.cc:260] T 1730de82547a4ccca398090e71d75807 P c2e6c5753e2546d9bc00393d25672551 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2e6c5753e2546d9bc00393d25672551" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 39179 } } peers { permanent_uuid: "bf7cab7961534d01af401cdd0fc4c5ff" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 36763 } } peers { permanent_uuid: "b7427e659a2b4fff8dc2e523e45d0718" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43529 } }
I20260925 15:13:44.008005 20393 ts_tablet_manager.cc:1428] T 1730de82547a4ccca398090e71d75807 P c2e6c5753e2546d9bc00393d25672551: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 15:13:44.153110 20399 raft_consensus.cc:482] T 1730de82547a4ccca398090e71d75807 P b7427e659a2b4fff8dc2e523e45d0718 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:13:44.153334 20399 raft_consensus.cc:504] T 1730de82547a4ccca398090e71d75807 P b7427e659a2b4fff8dc2e523e45d0718 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2e6c5753e2546d9bc00393d25672551" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 39179 } } peers { permanent_uuid: "bf7cab7961534d01af401cdd0fc4c5ff" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 36763 } } peers { permanent_uuid: "b7427e659a2b4fff8dc2e523e45d0718" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43529 } }
I20260925 15:13:44.155112 20399 leader_election.cc:288] T 1730de82547a4ccca398090e71d75807 P b7427e659a2b4fff8dc2e523e45d0718 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2e6c5753e2546d9bc00393d25672551 (127.12.8.1:39179), bf7cab7961534d01af401cdd0fc4c5ff (127.12.8.2:36763)
I20260925 15:13:44.169552 20392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:44.155902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.3:42565 (local address 127.12.8.2:36763)
0925 15:13:44.156097 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:44.156104 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:44.163299 (+ 7195us) server_negotiation.cc:407] Connection header received
0925 15:13:44.163581 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:44.163588 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:44.163672 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:44.163768 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:13:44.164389 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:44.165211 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:44.166975 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:44.168315 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:44.168480 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:44.168574 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:44.169160 (+ 586us) server_negotiation.cc:299] Negotiation successful
0925 15:13:44.169278 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 15:13:44.170387 20460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:44.155510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.2:36763 (local address 127.12.8.3:42565)
0925 15:13:44.163202 (+ 7692us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:44.163219 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:44.163411 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:44.163825 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:44.163830 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:44.163855 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:13:44.164161 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:44.164172 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:44.165340 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:44.165347 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:44.166804 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:44.166815 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:44.168670 (+ 1855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:44.168678 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:44.168923 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:44.169019 (+ 96us) client_negotiation.cc:769] Sending connection context
0925 15:13:44.169826 (+ 807us) client_negotiation.cc:240] Negotiation successful
0925 15:13:44.169948 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7566,"mutex_wait_us":65,"thread_start_us":6681,"threads_started":1}
I20260925 15:13:44.170383 20462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:44.155664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.3:57717 (local address 127.12.8.1:39179)
0925 15:13:44.162040 (+ 6376us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:44.162048 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:44.162792 (+ 744us) server_negotiation.cc:407] Connection header received
0925 15:13:44.162874 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:44.162878 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:44.162938 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:44.163044 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:13:44.163533 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:44.164624 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:44.167154 (+ 2530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:44.168377 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:44.168481 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:44.168577 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:44.169896 (+ 1319us) server_negotiation.cc:299] Negotiation successful
0925 15:13:44.170024 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6215,"thread_start_us":6086,"threads_started":1}
I20260925 15:13:44.170840 20461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:44.155289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.1:39179 (local address 127.12.8.3:57717)
0925 15:13:44.162533 (+ 7244us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:44.162559 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:44.162698 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:44.163064 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:44.163069 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:44.163094 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:13:44.163340 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:44.163350 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:44.164761 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:44.164766 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:44.166996 (+ 2230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:44.167010 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:44.168712 (+ 1702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:44.168717 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:44.168923 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:44.169010 (+ 87us) client_negotiation.cc:769] Sending connection context
0925 15:13:44.169102 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 15:13:44.169247 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7027,"mutex_wait_us":75,"thread_start_us":7252,"threads_started":1}
I20260925 15:13:44.171290 20185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1730de82547a4ccca398090e71d75807" candidate_uuid: "b7427e659a2b4fff8dc2e523e45d0718" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf7cab7961534d01af401cdd0fc4c5ff" is_pre_election: true
I20260925 15:13:44.171626 20045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1730de82547a4ccca398090e71d75807" candidate_uuid: "b7427e659a2b4fff8dc2e523e45d0718" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2e6c5753e2546d9bc00393d25672551" is_pre_election: true
I20260925 15:13:44.171768 20185 raft_consensus.cc:2454] T 1730de82547a4ccca398090e71d75807 P bf7cab7961534d01af401cdd0fc4c5ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7427e659a2b4fff8dc2e523e45d0718 in term 0.
I20260925 15:13:44.171869 20045 raft_consensus.cc:2454] T 1730de82547a4ccca398090e71d75807 P c2e6c5753e2546d9bc00393d25672551 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7427e659a2b4fff8dc2e523e45d0718 in term 0.
I20260925 15:13:44.172389 20251 leader_election.cc:302] T 1730de82547a4ccca398090e71d75807 P b7427e659a2b4fff8dc2e523e45d0718 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: b7427e659a2b4fff8dc2e523e45d0718, bf7cab7961534d01af401cdd0fc4c5ff, c2e6c5753e2546d9bc00393d25672551; no voters:
I20260925 15:13:44.172783 20399 raft_consensus.cc:2781] T 1730de82547a4ccca398090e71d75807 P b7427e659a2b4fff8dc2e523e45d0718 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:13:44.172910 20399 raft_consensus.cc:482] T 1730de82547a4ccca398090e71d75807 P b7427e659a2b4fff8dc2e523e45d0718 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:13:44.173015 20399 raft_consensus.cc:3037] T 1730de82547a4ccca398090e71d75807 P b7427e659a2b4fff8dc2e523e45d0718 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:13:44.175987 20399 raft_consensus.cc:504] T 1730de82547a4ccca398090e71d75807 P b7427e659a2b4fff8dc2e523e45d0718 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2e6c5753e2546d9bc00393d25672551" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 39179 } } peers { permanent_uuid: "bf7cab7961534d01af401cdd0fc4c5ff" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 36763 } } peers { permanent_uuid: "b7427e659a2b4fff8dc2e523e45d0718" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43529 } }
I20260925 15:13:44.176597 20399 leader_election.cc:288] T 1730de82547a4ccca398090e71d75807 P b7427e659a2b4fff8dc2e523e45d0718 [CANDIDATE]: Term 1 election: Requested vote from peers c2e6c5753e2546d9bc00393d25672551 (127.12.8.1:39179), bf7cab7961534d01af401cdd0fc4c5ff (127.12.8.2:36763)
I20260925 15:13:44.176951 20045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1730de82547a4ccca398090e71d75807" candidate_uuid: "b7427e659a2b4fff8dc2e523e45d0718" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2e6c5753e2546d9bc00393d25672551"
I20260925 15:13:44.177028 20185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1730de82547a4ccca398090e71d75807" candidate_uuid: "b7427e659a2b4fff8dc2e523e45d0718" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf7cab7961534d01af401cdd0fc4c5ff"
I20260925 15:13:44.177129 20045 raft_consensus.cc:3037] T 1730de82547a4ccca398090e71d75807 P c2e6c5753e2546d9bc00393d25672551 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:13:44.177248 20185 raft_consensus.cc:3037] T 1730de82547a4ccca398090e71d75807 P bf7cab7961534d01af401cdd0fc4c5ff [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:13:44.179654 20045 raft_consensus.cc:2454] T 1730de82547a4ccca398090e71d75807 P c2e6c5753e2546d9bc00393d25672551 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7427e659a2b4fff8dc2e523e45d0718 in term 1.
I20260925 15:13:44.180083 20252 leader_election.cc:302] T 1730de82547a4ccca398090e71d75807 P b7427e659a2b4fff8dc2e523e45d0718 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7427e659a2b4fff8dc2e523e45d0718, c2e6c5753e2546d9bc00393d25672551; no voters:
I20260925 15:13:44.180387 20399 raft_consensus.cc:2781] T 1730de82547a4ccca398090e71d75807 P b7427e659a2b4fff8dc2e523e45d0718 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:13:44.180456 20185 raft_consensus.cc:2454] T 1730de82547a4ccca398090e71d75807 P bf7cab7961534d01af401cdd0fc4c5ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7427e659a2b4fff8dc2e523e45d0718 in term 1.
I20260925 15:13:44.181236 20399 raft_consensus.cc:686] T 1730de82547a4ccca398090e71d75807 P b7427e659a2b4fff8dc2e523e45d0718 [term 1 LEADER]: Becoming Leader. State: Replica: b7427e659a2b4fff8dc2e523e45d0718, State: Running, Role: LEADER
I20260925 15:13:44.181651 20399 consensus_queue.cc:237] T 1730de82547a4ccca398090e71d75807 P b7427e659a2b4fff8dc2e523e45d0718 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2e6c5753e2546d9bc00393d25672551" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 39179 } } peers { permanent_uuid: "bf7cab7961534d01af401cdd0fc4c5ff" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 36763 } } peers { permanent_uuid: "b7427e659a2b4fff8dc2e523e45d0718" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43529 } }
I20260925 15:13:44.185153 19882 catalog_manager.cc:5495] T 1730de82547a4ccca398090e71d75807 P b7427e659a2b4fff8dc2e523e45d0718 reported cstate change: term changed from 0 to 1, leader changed from <none> to b7427e659a2b4fff8dc2e523e45d0718 (127.12.8.3). New cstate: current_term: 1 leader_uuid: "b7427e659a2b4fff8dc2e523e45d0718" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2e6c5753e2546d9bc00393d25672551" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 39179 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf7cab7961534d01af401cdd0fc4c5ff" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 36763 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b7427e659a2b4fff8dc2e523e45d0718" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43529 } health_report { overall_health: HEALTHY } } }
I20260925 15:13:44.239269 20466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:44.233538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52996 (local address 127.12.8.3:43529)
0925 15:13:44.234092 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:44.234096 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:44.234114 (+ 18us) server_negotiation.cc:407] Connection header received
0925 15:13:44.234251 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:44.234256 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:44.234340 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:44.234513 (+ 173us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:13:44.234918 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:44.235677 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:44.236966 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:44.237390 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:44.237522 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:44.237972 (+ 450us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:13:44.238629 (+ 657us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:13:44.238779 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:44.238879 (+ 100us) server_negotiation.cc:299] Negotiation successful
0925 15:13:44.238955 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":276,"threads_started":1}
I20260925 15:13:44.243028 20045 raft_consensus.cc:1261] T 1730de82547a4ccca398090e71d75807 P c2e6c5753e2546d9bc00393d25672551 [term 1 FOLLOWER]: Refusing update from remote peer b7427e659a2b4fff8dc2e523e45d0718: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:13:44.243279 20185 raft_consensus.cc:1261] T 1730de82547a4ccca398090e71d75807 P bf7cab7961534d01af401cdd0fc4c5ff [term 1 FOLLOWER]: Refusing update from remote peer b7427e659a2b4fff8dc2e523e45d0718: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:13:44.243676 20463 consensus_queue.cc:1035] T 1730de82547a4ccca398090e71d75807 P b7427e659a2b4fff8dc2e523e45d0718 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2e6c5753e2546d9bc00393d25672551" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 39179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:13:44.244040 20399 consensus_queue.cc:1035] T 1730de82547a4ccca398090e71d75807 P b7427e659a2b4fff8dc2e523e45d0718 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf7cab7961534d01af401cdd0fc4c5ff" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 36763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:13:44.265110 19883 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59816:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:13:44.265410 19883 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59816:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:13:44.269209 19883 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3312e9eb760f4482923279ed63ae02a8: Sending DeleteTablet for 3 replicas of tablet 1730de82547a4ccca398090e71d75807
I20260925 15:13:44.269958 20021 tablet_service.cc:1514] Processing DeleteTablet for tablet 1730de82547a4ccca398090e71d75807 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:13:44 UTC) from {username='oryx', principal='oryx/127.12.8.62@KRBTEST.COM'} at 127.0.0.1:56062
I20260925 15:13:44.270267 20417 tablet_replica.cc:330] T 1730de82547a4ccca398090e71d75807 P c2e6c5753e2546d9bc00393d25672551: stopping tablet replica
I20260925 15:13:44.270462 20417 raft_consensus.cc:2229] T 1730de82547a4ccca398090e71d75807 P c2e6c5753e2546d9bc00393d25672551 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:13:44.270685 20417 raft_consensus.cc:2258] T 1730de82547a4ccca398090e71d75807 P c2e6c5753e2546d9bc00393d25672551 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:13:44.271672 20417 ts_tablet_manager.cc:1905] T 1730de82547a4ccca398090e71d75807 P c2e6c5753e2546d9bc00393d25672551: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:13:44.279083 20160 tablet_service.cc:1514] Processing DeleteTablet for tablet 1730de82547a4ccca398090e71d75807 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:13:44 UTC) from {username='oryx', principal='oryx/127.12.8.62@KRBTEST.COM'} at 127.0.0.1:33624
I20260925 15:13:44.279482 20418 tablet_replica.cc:330] T 1730de82547a4ccca398090e71d75807 P bf7cab7961534d01af401cdd0fc4c5ff: stopping tablet replica
I20260925 15:13:44.279585 20299 tablet_service.cc:1514] Processing DeleteTablet for tablet 1730de82547a4ccca398090e71d75807 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:13:44 UTC) from {username='oryx', principal='oryx/127.12.8.62@KRBTEST.COM'} at 127.0.0.1:56158
I20260925 15:13:44.279678 20418 raft_consensus.cc:2229] T 1730de82547a4ccca398090e71d75807 P bf7cab7961534d01af401cdd0fc4c5ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:13:44.279907 20416 tablet_replica.cc:330] T 1730de82547a4ccca398090e71d75807 P b7427e659a2b4fff8dc2e523e45d0718: stopping tablet replica
I20260925 15:13:44.279963 20418 raft_consensus.cc:2258] T 1730de82547a4ccca398090e71d75807 P bf7cab7961534d01af401cdd0fc4c5ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:13:44.280090 20416 raft_consensus.cc:2229] T 1730de82547a4ccca398090e71d75807 P b7427e659a2b4fff8dc2e523e45d0718 [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:13:44.281400 20418 ts_tablet_manager.cc:1905] T 1730de82547a4ccca398090e71d75807 P bf7cab7961534d01af401cdd0fc4c5ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:13:44.282066 19882 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59816:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:13:44.282783 19882 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:13:44.288781 20418 ts_tablet_manager.cc:1918] T 1730de82547a4ccca398090e71d75807 P bf7cab7961534d01af401cdd0fc4c5ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:13:44.289019 20418 log.cc:1196] T 1730de82547a4ccca398090e71d75807 P bf7cab7961534d01af401cdd0fc4c5ff: Deleting WAL directory at /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/minicluster-data/ts-1/wal/wals/1730de82547a4ccca398090e71d75807
I20260925 15:13:44.289523 20418 ts_tablet_manager.cc:1939] T 1730de82547a4ccca398090e71d75807 P bf7cab7961534d01af401cdd0fc4c5ff: Deleting consensus metadata
I20260925 15:13:44.290737 19869 catalog_manager.cc:4841] TS bf7cab7961534d01af401cdd0fc4c5ff (127.12.8.2:36763): tablet 1730de82547a4ccca398090e71d75807 (table test-table [id=9cadef999c5a4e5498427a8147abb730]) successfully deleted
I20260925 15:13:44.291124 20160 tablet_service.cc:1467] Processing CreateTablet for tablet b6097868c769457c8e6f90322a06555a (DEFAULT_TABLE table=test-table [id=1e02d9bb7e9b4d468d709ddfefb5a2e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:13:44.291529 20160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6097868c769457c8e6f90322a06555a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:13:44.291462 20299 tablet_service.cc:1467] Processing CreateTablet for tablet b6097868c769457c8e6f90322a06555a (DEFAULT_TABLE table=test-table [id=1e02d9bb7e9b4d468d709ddfefb5a2e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:13:44.291883 20299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6097868c769457c8e6f90322a06555a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:13:44.298349 20394 tablet_bootstrap.cc:492] T b6097868c769457c8e6f90322a06555a P bf7cab7961534d01af401cdd0fc4c5ff: Bootstrap starting.
I20260925 15:13:44.300171 20394 tablet_bootstrap.cc:654] T b6097868c769457c8e6f90322a06555a P bf7cab7961534d01af401cdd0fc4c5ff: Neither blocks nor log segments found. Creating new log.
I20260925 15:13:44.302922 20021 tablet_service.cc:1467] Processing CreateTablet for tablet b6097868c769457c8e6f90322a06555a (DEFAULT_TABLE table=test-table [id=1e02d9bb7e9b4d468d709ddfefb5a2e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:13:44.303784 20021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6097868c769457c8e6f90322a06555a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:13:44.308693 20393 tablet_bootstrap.cc:492] T b6097868c769457c8e6f90322a06555a P c2e6c5753e2546d9bc00393d25672551: Bootstrap starting.
I20260925 15:13:44.308827 20417 ts_tablet_manager.cc:1918] T 1730de82547a4ccca398090e71d75807 P c2e6c5753e2546d9bc00393d25672551: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:13:44.308956 20417 log.cc:1196] T 1730de82547a4ccca398090e71d75807 P c2e6c5753e2546d9bc00393d25672551: Deleting WAL directory at /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/minicluster-data/ts-0/wal/wals/1730de82547a4ccca398090e71d75807
I20260925 15:13:44.309422 20417 ts_tablet_manager.cc:1939] T 1730de82547a4ccca398090e71d75807 P c2e6c5753e2546d9bc00393d25672551: Deleting consensus metadata
I20260925 15:13:44.310420 20393 tablet_bootstrap.cc:654] T b6097868c769457c8e6f90322a06555a P c2e6c5753e2546d9bc00393d25672551: Neither blocks nor log segments found. Creating new log.
I20260925 15:13:44.312763 20394 tablet_bootstrap.cc:492] T b6097868c769457c8e6f90322a06555a P bf7cab7961534d01af401cdd0fc4c5ff: No bootstrap required, opened a new log
I20260925 15:13:44.313052 20394 ts_tablet_manager.cc:1397] T b6097868c769457c8e6f90322a06555a P bf7cab7961534d01af401cdd0fc4c5ff: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.006s
I20260925 15:13:44.314021 20394 raft_consensus.cc:348] T b6097868c769457c8e6f90322a06555a P bf7cab7961534d01af401cdd0fc4c5ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2e6c5753e2546d9bc00393d25672551" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 39179 } } peers { permanent_uuid: "bf7cab7961534d01af401cdd0fc4c5ff" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 36763 } } peers { permanent_uuid: "b7427e659a2b4fff8dc2e523e45d0718" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43529 } }
I20260925 15:13:44.314412 20394 raft_consensus.cc:374] T b6097868c769457c8e6f90322a06555a P bf7cab7961534d01af401cdd0fc4c5ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:13:44.314538 20394 raft_consensus.cc:729] T b6097868c769457c8e6f90322a06555a P bf7cab7961534d01af401cdd0fc4c5ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf7cab7961534d01af401cdd0fc4c5ff, State: Initialized, Role: FOLLOWER
I20260925 15:13:44.314900 20394 consensus_queue.cc:260] T b6097868c769457c8e6f90322a06555a P bf7cab7961534d01af401cdd0fc4c5ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2e6c5753e2546d9bc00393d25672551" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 39179 } } peers { permanent_uuid: "bf7cab7961534d01af401cdd0fc4c5ff" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 36763 } } peers { permanent_uuid: "b7427e659a2b4fff8dc2e523e45d0718" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43529 } }
I20260925 15:13:44.317642 20394 ts_tablet_manager.cc:1428] T b6097868c769457c8e6f90322a06555a P bf7cab7961534d01af401cdd0fc4c5ff: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260925 15:13:44.318921 20395 tablet_bootstrap.cc:492] T b6097868c769457c8e6f90322a06555a P b7427e659a2b4fff8dc2e523e45d0718: Bootstrap starting.
I20260925 15:13:44.319059 19870 catalog_manager.cc:4841] TS c2e6c5753e2546d9bc00393d25672551 (127.12.8.1:39179): tablet 1730de82547a4ccca398090e71d75807 (table test-table [id=9cadef999c5a4e5498427a8147abb730]) successfully deleted
I20260925 15:13:44.320678 20395 tablet_bootstrap.cc:654] T b6097868c769457c8e6f90322a06555a P b7427e659a2b4fff8dc2e523e45d0718: Neither blocks nor log segments found. Creating new log.
I20260925 15:13:44.325560 20393 tablet_bootstrap.cc:492] T b6097868c769457c8e6f90322a06555a P c2e6c5753e2546d9bc00393d25672551: No bootstrap required, opened a new log
I20260925 15:13:44.325737 20393 ts_tablet_manager.cc:1397] T b6097868c769457c8e6f90322a06555a P c2e6c5753e2546d9bc00393d25672551: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.005s
I20260925 15:13:44.326669 20416 raft_consensus.cc:2258] T 1730de82547a4ccca398090e71d75807 P b7427e659a2b4fff8dc2e523e45d0718 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:13:44.326576 20393 raft_consensus.cc:348] T b6097868c769457c8e6f90322a06555a P c2e6c5753e2546d9bc00393d25672551 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2e6c5753e2546d9bc00393d25672551" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 39179 } } peers { permanent_uuid: "bf7cab7961534d01af401cdd0fc4c5ff" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 36763 } } peers { permanent_uuid: "b7427e659a2b4fff8dc2e523e45d0718" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43529 } }
I20260925 15:13:44.326818 20393 raft_consensus.cc:374] T b6097868c769457c8e6f90322a06555a P c2e6c5753e2546d9bc00393d25672551 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:13:44.326915 20393 raft_consensus.cc:729] T b6097868c769457c8e6f90322a06555a P c2e6c5753e2546d9bc00393d25672551 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2e6c5753e2546d9bc00393d25672551, State: Initialized, Role: FOLLOWER
I20260925 15:13:44.327172 20393 consensus_queue.cc:260] T b6097868c769457c8e6f90322a06555a P c2e6c5753e2546d9bc00393d25672551 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2e6c5753e2546d9bc00393d25672551" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 39179 } } peers { permanent_uuid: "bf7cab7961534d01af401cdd0fc4c5ff" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 36763 } } peers { permanent_uuid: "b7427e659a2b4fff8dc2e523e45d0718" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43529 } }
I20260925 15:13:44.328140 20416 ts_tablet_manager.cc:1905] T 1730de82547a4ccca398090e71d75807 P b7427e659a2b4fff8dc2e523e45d0718: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:13:44.330822 20393 ts_tablet_manager.cc:1428] T b6097868c769457c8e6f90322a06555a P c2e6c5753e2546d9bc00393d25672551: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260925 15:13:44.335341 20416 ts_tablet_manager.cc:1918] T 1730de82547a4ccca398090e71d75807 P b7427e659a2b4fff8dc2e523e45d0718: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:13:44.335492 20416 log.cc:1196] T 1730de82547a4ccca398090e71d75807 P b7427e659a2b4fff8dc2e523e45d0718: Deleting WAL directory at /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/minicluster-data/ts-2/wal/wals/1730de82547a4ccca398090e71d75807
I20260925 15:13:44.335947 20416 ts_tablet_manager.cc:1939] T 1730de82547a4ccca398090e71d75807 P b7427e659a2b4fff8dc2e523e45d0718: Deleting consensus metadata
I20260925 15:13:44.337167 19869 catalog_manager.cc:4841] TS b7427e659a2b4fff8dc2e523e45d0718 (127.12.8.3:43529): tablet 1730de82547a4ccca398090e71d75807 (table test-table [id=9cadef999c5a4e5498427a8147abb730]) successfully deleted
I20260925 15:13:44.340889 20395 tablet_bootstrap.cc:492] T b6097868c769457c8e6f90322a06555a P b7427e659a2b4fff8dc2e523e45d0718: No bootstrap required, opened a new log
I20260925 15:13:44.341066 20395 ts_tablet_manager.cc:1397] T b6097868c769457c8e6f90322a06555a P b7427e659a2b4fff8dc2e523e45d0718: Time spent bootstrapping tablet: real 0.022s user 0.001s sys 0.008s
I20260925 15:13:44.341954 20395 raft_consensus.cc:348] T b6097868c769457c8e6f90322a06555a P b7427e659a2b4fff8dc2e523e45d0718 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2e6c5753e2546d9bc00393d25672551" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 39179 } } peers { permanent_uuid: "bf7cab7961534d01af401cdd0fc4c5ff" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 36763 } } peers { permanent_uuid: "b7427e659a2b4fff8dc2e523e45d0718" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43529 } }
I20260925 15:13:44.342232 20395 raft_consensus.cc:374] T b6097868c769457c8e6f90322a06555a P b7427e659a2b4fff8dc2e523e45d0718 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:13:44.342324 20395 raft_consensus.cc:729] T b6097868c769457c8e6f90322a06555a P b7427e659a2b4fff8dc2e523e45d0718 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7427e659a2b4fff8dc2e523e45d0718, State: Initialized, Role: FOLLOWER
I20260925 15:13:44.342597 20395 consensus_queue.cc:260] T b6097868c769457c8e6f90322a06555a P b7427e659a2b4fff8dc2e523e45d0718 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2e6c5753e2546d9bc00393d25672551" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 39179 } } peers { permanent_uuid: "bf7cab7961534d01af401cdd0fc4c5ff" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 36763 } } peers { permanent_uuid: "b7427e659a2b4fff8dc2e523e45d0718" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43529 } }
I20260925 15:13:44.343106 20395 ts_tablet_manager.cc:1428] T b6097868c769457c8e6f90322a06555a P b7427e659a2b4fff8dc2e523e45d0718: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 15:13:44.586293 20469 raft_consensus.cc:482] T b6097868c769457c8e6f90322a06555a P b7427e659a2b4fff8dc2e523e45d0718 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:13:44.586550 20469 raft_consensus.cc:504] T b6097868c769457c8e6f90322a06555a P b7427e659a2b4fff8dc2e523e45d0718 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2e6c5753e2546d9bc00393d25672551" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 39179 } } peers { permanent_uuid: "bf7cab7961534d01af401cdd0fc4c5ff" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 36763 } } peers { permanent_uuid: "b7427e659a2b4fff8dc2e523e45d0718" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43529 } }
I20260925 15:13:44.587538 20469 leader_election.cc:288] T b6097868c769457c8e6f90322a06555a P b7427e659a2b4fff8dc2e523e45d0718 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2e6c5753e2546d9bc00393d25672551 (127.12.8.1:39179), bf7cab7961534d01af401cdd0fc4c5ff (127.12.8.2:36763)
I20260925 15:13:44.588097 20185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6097868c769457c8e6f90322a06555a" candidate_uuid: "b7427e659a2b4fff8dc2e523e45d0718" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf7cab7961534d01af401cdd0fc4c5ff" is_pre_election: true
I20260925 15:13:44.588150 20045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6097868c769457c8e6f90322a06555a" candidate_uuid: "b7427e659a2b4fff8dc2e523e45d0718" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2e6c5753e2546d9bc00393d25672551" is_pre_election: true
I20260925 15:13:44.588346 20185 raft_consensus.cc:2454] T b6097868c769457c8e6f90322a06555a P bf7cab7961534d01af401cdd0fc4c5ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7427e659a2b4fff8dc2e523e45d0718 in term 0.
I20260925 15:13:44.588418 20045 raft_consensus.cc:2454] T b6097868c769457c8e6f90322a06555a P c2e6c5753e2546d9bc00393d25672551 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7427e659a2b4fff8dc2e523e45d0718 in term 0.
I20260925 15:13:44.588815 20251 leader_election.cc:302] T b6097868c769457c8e6f90322a06555a P b7427e659a2b4fff8dc2e523e45d0718 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7427e659a2b4fff8dc2e523e45d0718, bf7cab7961534d01af401cdd0fc4c5ff; no voters:
I20260925 15:13:44.589154 20469 raft_consensus.cc:2781] T b6097868c769457c8e6f90322a06555a P b7427e659a2b4fff8dc2e523e45d0718 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:13:44.589287 20469 raft_consensus.cc:482] T b6097868c769457c8e6f90322a06555a P b7427e659a2b4fff8dc2e523e45d0718 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:13:44.589393 20469 raft_consensus.cc:3037] T b6097868c769457c8e6f90322a06555a P b7427e659a2b4fff8dc2e523e45d0718 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:13:44.592824 20469 raft_consensus.cc:504] T b6097868c769457c8e6f90322a06555a P b7427e659a2b4fff8dc2e523e45d0718 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2e6c5753e2546d9bc00393d25672551" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 39179 } } peers { permanent_uuid: "bf7cab7961534d01af401cdd0fc4c5ff" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 36763 } } peers { permanent_uuid: "b7427e659a2b4fff8dc2e523e45d0718" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43529 } }
I20260925 15:13:44.593526 20469 leader_election.cc:288] T b6097868c769457c8e6f90322a06555a P b7427e659a2b4fff8dc2e523e45d0718 [CANDIDATE]: Term 1 election: Requested vote from peers c2e6c5753e2546d9bc00393d25672551 (127.12.8.1:39179), bf7cab7961534d01af401cdd0fc4c5ff (127.12.8.2:36763)
I20260925 15:13:44.593889 20045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6097868c769457c8e6f90322a06555a" candidate_uuid: "b7427e659a2b4fff8dc2e523e45d0718" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2e6c5753e2546d9bc00393d25672551"
I20260925 15:13:44.594098 20045 raft_consensus.cc:3037] T b6097868c769457c8e6f90322a06555a P c2e6c5753e2546d9bc00393d25672551 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:13:44.594213 20185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6097868c769457c8e6f90322a06555a" candidate_uuid: "b7427e659a2b4fff8dc2e523e45d0718" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf7cab7961534d01af401cdd0fc4c5ff"
I20260925 15:13:44.594420 20185 raft_consensus.cc:3037] T b6097868c769457c8e6f90322a06555a P bf7cab7961534d01af401cdd0fc4c5ff [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:13:44.596352 20185 raft_consensus.cc:2454] T b6097868c769457c8e6f90322a06555a P bf7cab7961534d01af401cdd0fc4c5ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7427e659a2b4fff8dc2e523e45d0718 in term 1.
I20260925 15:13:44.596735 20251 leader_election.cc:302] T b6097868c769457c8e6f90322a06555a P b7427e659a2b4fff8dc2e523e45d0718 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7427e659a2b4fff8dc2e523e45d0718, bf7cab7961534d01af401cdd0fc4c5ff; no voters:
I20260925 15:13:44.597049 20045 raft_consensus.cc:2454] T b6097868c769457c8e6f90322a06555a P c2e6c5753e2546d9bc00393d25672551 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7427e659a2b4fff8dc2e523e45d0718 in term 1.
I20260925 15:13:44.597074 20469 raft_consensus.cc:2781] T b6097868c769457c8e6f90322a06555a P b7427e659a2b4fff8dc2e523e45d0718 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:13:44.597240 20469 raft_consensus.cc:686] T b6097868c769457c8e6f90322a06555a P b7427e659a2b4fff8dc2e523e45d0718 [term 1 LEADER]: Becoming Leader. State: Replica: b7427e659a2b4fff8dc2e523e45d0718, State: Running, Role: LEADER
I20260925 15:13:44.597584 20469 consensus_queue.cc:237] T b6097868c769457c8e6f90322a06555a P b7427e659a2b4fff8dc2e523e45d0718 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2e6c5753e2546d9bc00393d25672551" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 39179 } } peers { permanent_uuid: "bf7cab7961534d01af401cdd0fc4c5ff" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 36763 } } peers { permanent_uuid: "b7427e659a2b4fff8dc2e523e45d0718" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43529 } }
I20260925 15:13:44.600391 19883 catalog_manager.cc:5495] T b6097868c769457c8e6f90322a06555a P b7427e659a2b4fff8dc2e523e45d0718 reported cstate change: term changed from 0 to 1, leader changed from <none> to b7427e659a2b4fff8dc2e523e45d0718 (127.12.8.3). New cstate: current_term: 1 leader_uuid: "b7427e659a2b4fff8dc2e523e45d0718" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2e6c5753e2546d9bc00393d25672551" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 39179 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf7cab7961534d01af401cdd0fc4c5ff" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 36763 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b7427e659a2b4fff8dc2e523e45d0718" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 43529 } health_report { overall_health: HEALTHY } } }
I20260925 15:13:44.701391 20479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:44.691677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.3:34141 (local address 127.12.8.2:36763)
0925 15:13:44.695958 (+ 4281us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:44.695964 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:44.696356 (+ 392us) server_negotiation.cc:407] Connection header received
0925 15:13:44.696529 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:44.696533 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:44.696594 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:44.696672 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:13:44.697128 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:44.697949 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:44.699844 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:44.700210 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:44.700304 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:44.700771 (+ 467us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:13:44.700878 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:13:44.700942 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:44.701093 (+ 151us) server_negotiation.cc:299] Negotiation successful
0925 15:13:44.701151 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4146,"thread_start_us":3991,"threads_started":1}
I20260925 15:13:44.701558 20480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:44.691693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.2:36763 (local address 127.12.8.3:34141)
0925 15:13:44.696239 (+ 4546us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:44.696260 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:44.696401 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:44.696708 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:44.696712 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:44.696723 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 15:13:44.696972 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:44.696982 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:44.698073 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:44.698078 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:44.699710 (+ 1632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:44.699724 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:44.700346 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:44.700351 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:44.700544 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:44.700607 (+ 63us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 15:13:44.700962 (+ 355us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 15:13:44.700972 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 15:13:44.701047 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 15:13:44.701186 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4369,"thread_start_us":4195,"threads_started":1}
I20260925 15:13:44.708801 20185 raft_consensus.cc:1261] T b6097868c769457c8e6f90322a06555a P bf7cab7961534d01af401cdd0fc4c5ff [term 1 FOLLOWER]: Refusing update from remote peer b7427e659a2b4fff8dc2e523e45d0718: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:13:44.708844 20045 raft_consensus.cc:1261] T b6097868c769457c8e6f90322a06555a P c2e6c5753e2546d9bc00393d25672551 [term 1 FOLLOWER]: Refusing update from remote peer b7427e659a2b4fff8dc2e523e45d0718: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:13:44.709296 20469 consensus_queue.cc:1035] T b6097868c769457c8e6f90322a06555a P b7427e659a2b4fff8dc2e523e45d0718 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2e6c5753e2546d9bc00393d25672551" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 39179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:13:44.709591 20463 consensus_queue.cc:1035] T b6097868c769457c8e6f90322a06555a P b7427e659a2b4fff8dc2e523e45d0718 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf7cab7961534d01af401cdd0fc4c5ff" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 36763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:13:44.734025 20466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:44.728750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.2:47547 (local address 127.12.8.3:43529)
0925 15:13:44.728915 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:44.728923 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:44.729346 (+ 423us) server_negotiation.cc:407] Connection header received
0925 15:13:44.729546 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:44.729551 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:44.729629 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:44.729723 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:13:44.730094 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:44.730859 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:44.732439 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:44.733034 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:44.733135 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:44.733504 (+ 369us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:13:44.733595 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:13:44.733662 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:44.733776 (+ 114us) server_negotiation.cc:299] Negotiation successful
0925 15:13:44.733834 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 15:13:44.734122 20485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:44.728674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.3:43529 (local address 127.12.8.2:47547)
0925 15:13:44.729242 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:44.729261 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:44.729413 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:44.729750 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:44.729755 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:44.729768 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 15:13:44.729978 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:44.729986 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:44.730969 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:44.730972 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:44.732315 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:44.732325 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:44.733198 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:44.733203 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:44.733331 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:44.733365 (+ 34us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 15:13:44.733680 (+ 315us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 15:13:44.733687 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 15:13:44.733745 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 15:13:44.733846 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":289,"threads_started":1}
I20260925 15:13:44.761821 19882 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59816:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:13:44.762100 19882 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59816:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:13:44.764796 19882 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3312e9eb760f4482923279ed63ae02a8: Sending DeleteTablet for 3 replicas of tablet b6097868c769457c8e6f90322a06555a
I20260925 15:13:44.765520 20160 tablet_service.cc:1514] Processing DeleteTablet for tablet b6097868c769457c8e6f90322a06555a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:13:44 UTC) from {username='oryx', principal='oryx/127.12.8.62@KRBTEST.COM'} at 127.0.0.1:33624
I20260925 15:13:44.765527 20021 tablet_service.cc:1514] Processing DeleteTablet for tablet b6097868c769457c8e6f90322a06555a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:13:44 UTC) from {username='oryx', principal='oryx/127.12.8.62@KRBTEST.COM'} at 127.0.0.1:56062
I20260925 15:13:44.765566 20299 tablet_service.cc:1514] Processing DeleteTablet for tablet b6097868c769457c8e6f90322a06555a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:13:44 UTC) from {username='oryx', principal='oryx/127.12.8.62@KRBTEST.COM'} at 127.0.0.1:56158
I20260925 15:13:44.765817 20416 tablet_replica.cc:330] T b6097868c769457c8e6f90322a06555a P b7427e659a2b4fff8dc2e523e45d0718: stopping tablet replica
I20260925 15:13:44.765807 20418 tablet_replica.cc:330] T b6097868c769457c8e6f90322a06555a P bf7cab7961534d01af401cdd0fc4c5ff: stopping tablet replica
I20260925 15:13:44.765960 20416 raft_consensus.cc:2229] T b6097868c769457c8e6f90322a06555a P b7427e659a2b4fff8dc2e523e45d0718 [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:13:44.766000 20418 raft_consensus.cc:2229] T b6097868c769457c8e6f90322a06555a P bf7cab7961534d01af401cdd0fc4c5ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:13:44.766212 20418 raft_consensus.cc:2258] T b6097868c769457c8e6f90322a06555a P bf7cab7961534d01af401cdd0fc4c5ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:13:44.766252 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 19955
I20260925 15:13:44.766270 20416 raft_consensus.cc:2258] T b6097868c769457c8e6f90322a06555a P b7427e659a2b4fff8dc2e523e45d0718 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:13:44.766485 20417 tablet_replica.cc:330] T b6097868c769457c8e6f90322a06555a P c2e6c5753e2546d9bc00393d25672551: stopping tablet replica
I20260925 15:13:44.766687 20417 raft_consensus.cc:2229] T b6097868c769457c8e6f90322a06555a P c2e6c5753e2546d9bc00393d25672551 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:13:44.766862 20417 raft_consensus.cc:2258] T b6097868c769457c8e6f90322a06555a P c2e6c5753e2546d9bc00393d25672551 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:13:44.768002 20418 ts_tablet_manager.cc:1905] T b6097868c769457c8e6f90322a06555a P bf7cab7961534d01af401cdd0fc4c5ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:13:44.768028 20416 ts_tablet_manager.cc:1905] T b6097868c769457c8e6f90322a06555a P b7427e659a2b4fff8dc2e523e45d0718: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:13:44.768066 20417 ts_tablet_manager.cc:1905] T b6097868c769457c8e6f90322a06555a P c2e6c5753e2546d9bc00393d25672551: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:13:44.772585 20416 ts_tablet_manager.cc:1918] T b6097868c769457c8e6f90322a06555a P b7427e659a2b4fff8dc2e523e45d0718: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 15:13:44.772740 20416 log.cc:1196] T b6097868c769457c8e6f90322a06555a P b7427e659a2b4fff8dc2e523e45d0718: Deleting WAL directory at /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/minicluster-data/ts-2/wal/wals/b6097868c769457c8e6f90322a06555a
I20260925 15:13:44.773146 20416 ts_tablet_manager.cc:1939] T b6097868c769457c8e6f90322a06555a P b7427e659a2b4fff8dc2e523e45d0718: Deleting consensus metadata
I20260925 15:13:44.774325 19869 catalog_manager.cc:4841] TS b7427e659a2b4fff8dc2e523e45d0718 (127.12.8.3:43529): tablet b6097868c769457c8e6f90322a06555a (table test-table [id=1e02d9bb7e9b4d468d709ddfefb5a2e8]) successfully deleted
I20260925 15:13:44.774729 20418 ts_tablet_manager.cc:1918] T b6097868c769457c8e6f90322a06555a P bf7cab7961534d01af401cdd0fc4c5ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 15:13:44.774931 20418 log.cc:1196] T b6097868c769457c8e6f90322a06555a P bf7cab7961534d01af401cdd0fc4c5ff: Deleting WAL directory at /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/minicluster-data/ts-1/wal/wals/b6097868c769457c8e6f90322a06555a
I20260925 15:13:44.775388 20418 ts_tablet_manager.cc:1939] T b6097868c769457c8e6f90322a06555a P bf7cab7961534d01af401cdd0fc4c5ff: Deleting consensus metadata
I20260925 15:13:44.776652 19869 catalog_manager.cc:4841] TS bf7cab7961534d01af401cdd0fc4c5ff (127.12.8.2:36763): tablet b6097868c769457c8e6f90322a06555a (table test-table [id=1e02d9bb7e9b4d468d709ddfefb5a2e8]) successfully deleted
I20260925 15:13:44.778218 20417 ts_tablet_manager.cc:1918] T b6097868c769457c8e6f90322a06555a P c2e6c5753e2546d9bc00393d25672551: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 15:13:44.778412 20417 log.cc:1196] T b6097868c769457c8e6f90322a06555a P c2e6c5753e2546d9bc00393d25672551: Deleting WAL directory at /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349162370577-12320-0/minicluster-data/ts-0/wal/wals/b6097868c769457c8e6f90322a06555a
I20260925 15:13:44.778901 20417 ts_tablet_manager.cc:1939] T b6097868c769457c8e6f90322a06555a P c2e6c5753e2546d9bc00393d25672551: Deleting consensus metadata
I20260925 15:13:44.780620 19870 catalog_manager.cc:4841] TS c2e6c5753e2546d9bc00393d25672551 (127.12.8.1:39179): tablet b6097868c769457c8e6f90322a06555a (table test-table [id=1e02d9bb7e9b4d468d709ddfefb5a2e8]) successfully deleted
Sep 25 15:13:44 dist-test-slave-p468 krb5kdc[19839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.8.1@KRBTEST.COM
I20260925 15:13:44.793279 20488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:44.773504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59650 (local address 127.12.8.1:39179)
0925 15:13:44.780567 (+ 7063us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:44.780573 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:44.780590 (+ 17us) server_negotiation.cc:407] Connection header received
0925 15:13:44.780691 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:44.780697 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:44.780776 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:44.780881 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:44.781995 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:44.782760 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:44.783887 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:44.784289 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:44.784387 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:44.788060 (+ 3673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:44.788074 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:44.788090 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:44.788124 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:44.790800 (+ 2676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:44.791387 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:44.791394 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:44.791396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:44.791507 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:44.791808 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:44.791811 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:44.791813 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:44.792165 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:44.792488 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:44.792714 (+ 226us) server_negotiation.cc:299] Negotiation successful
0925 15:13:44.792913 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6941,"thread_start_us":6872,"threads_started":1}
I20260925 15:13:44.793920 20086 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:13:44.890369 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 19955
I20260925 15:13:44.908202 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 20095
Sep 25 15:13:44 dist-test-slave-p468 krb5kdc[19839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.8.2@KRBTEST.COM
I20260925 15:13:44.922502 20479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:44.908873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35708 (local address 127.12.8.2:36763)
0925 15:13:44.909028 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:44.909032 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:44.909085 (+ 53us) server_negotiation.cc:407] Connection header received
0925 15:13:44.909235 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:44.909239 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:44.909315 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:44.909399 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:44.910630 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:44.911311 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:44.912551 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:44.912965 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:44.913055 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:44.917022 (+ 3967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:44.917035 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:44.917050 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:44.917080 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:44.919734 (+ 2654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:44.920374 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:44.920383 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:44.920385 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:44.920513 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:44.920864 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:44.920870 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:44.920873 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:44.921199 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:44.921499 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:44.921808 (+ 309us) server_negotiation.cc:299] Negotiation successful
0925 15:13:44.922029 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 15:13:44.923149 20225 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:13:45.064545 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 20095
I20260925 15:13:45.089385 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 20234
Sep 25 15:13:45 dist-test-slave-p468 krb5kdc[19839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.8.3@KRBTEST.COM
I20260925 15:13:45.104756 20466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:45.090093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52998 (local address 127.12.8.3:43529)
0925 15:13:45.090306 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:45.090311 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:45.090330 (+ 19us) server_negotiation.cc:407] Connection header received
0925 15:13:45.090523 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:45.090527 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:45.090590 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:45.090683 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:45.091887 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:45.092671 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:45.094149 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:45.094644 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:45.094741 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:45.099275 (+ 4534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:45.099289 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:45.099304 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:45.099338 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:45.102088 (+ 2750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:45.102824 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:45.102831 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:45.102833 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:45.102973 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:45.103427 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:45.103430 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:45.103432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:45.103756 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:45.104000 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:45.104243 (+ 243us) server_negotiation.cc:299] Negotiation successful
0925 15:13:45.104399 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 15:13:45.105376 20364 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:13:45.230348 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 20234
I20260925 15:13:45.251333 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 19852
I20260925 15:13:45.252537 19924 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:13:45.366242 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 19852
2026-09-25T15:13:45Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (5636 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790349162370577-12320-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:13:45 dist-test-slave-p468 krb5kdc[20495](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:13:45 dist-test-slave-p468 krb5kdc[20495](info): set up 2 sockets
Sep 25 15:13:45 dist-test-slave-p468 krb5kdc[20495](info): commencing operation
krb5kdc: starting...
W20260925 15:13:46.467303 12320 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.001s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790349162370577-12320-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790349162370577-12320-0/krb5kdc/test-user.keytab.
Sep 25 15:13:46 dist-test-slave-p468 krb5kdc[20495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:13:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:13:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.8.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab.
Entry for principal kudu/127.12.8.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab.
I20260925 15:13:46.684962 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790349162370577-12320-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790349162370577-12320-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790349162370577-12320-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.62:36285
--webserver_interface=127.12.8.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab
--principal=kudu/127.12.8.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:39855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.62:36285
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:13:47.005352 20508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:13:47.005631 20508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:13:47.005823 20508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:13:47.012869 20508 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:13:47.012996 20508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:13:47.013082 20508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:13:47.013177 20508 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:13:47.013242 20508 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:13:47.022868 20508 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:39855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790349162370577-12320-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790349162370577-12320-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.8.62:36285
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.62:36285
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.8.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790349162370577-12320-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:13:47.024073 20508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:13:47.025471 20508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:13:47.037068 20514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:47.037146 20516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:47.037184 20513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:13:47.038301 20508 server_base.cc:1034] running on GCE node
I20260925 15:13:47.038978 20508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:13:47.040303 20508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:13:47.041492 20508 hybrid_clock.cc:648] HybridClock initialized: now 1790349227041439 us; error 53 us; skew 500 ppm
I20260925 15:13:47.041771 20508 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-25T15:13:47Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1744 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:13:47 dist-test-slave-p468 krb5kdc[20522](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:13:47 dist-test-slave-p468 krb5kdc[20522](info): set up 2 sockets
Sep 25 15:13:47 dist-test-slave-p468 krb5kdc[20522](info): commencing operation
krb5kdc: starting...
W20260925 15:13:48.211273 12320 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/krb5kdc/test-user.keytab.
Sep 25 15:13:48 dist-test-slave-p468 krb5kdc[20522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349228, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:13:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:13:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.8.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab.
Entry for principal kudu/127.12.8.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab.
I20260925 15:13:48.425935 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.62:43599
--webserver_interface=127.12.8.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab
--principal=kudu/127.12.8.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:39047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.62:43599
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:13:48.739717 20535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:13:48.739995 20535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:13:48.740087 20535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:13:48.747037 20535 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:13:48.747143 20535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:13:48.747188 20535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:13:48.747237 20535 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:13:48.747284 20535 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:13:48.756610 20535 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:39047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.8.62:43599
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab
--principal=kudu/127.12.8.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.62:43599
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.8.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:13:48.757752 20535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:13:48.759014 20535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:13:48.771054 20540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:48.771126 20543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:48.771150 20541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:13:48.775580 20535 server_base.cc:1034] running on GCE node
I20260925 15:13:48.776356 20535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:13:48.777712 20535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:13:48.778961 20535 hybrid_clock.cc:648] HybridClock initialized: now 1790349228778923 us; error 56 us; skew 500 ppm
I20260925 15:13:48.779328 20535 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:13:48 dist-test-slave-p468 krb5kdc[20522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349228, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:13:48.782429 20535 init.cc:377] Logged in from keytab as kudu/127.12.8.62@KRBTEST.COM (short username kudu)
I20260925 15:13:48.783895 20535 webserver.cc:460] Webserver started at http://127.12.8.62:42379/ using document root <none> and password file <none>
I20260925 15:13:48.784560 20535 fs_manager.cc:359] Metadata directory not provided
I20260925 15:13:48.784699 20535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:13:48.784997 20535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:13:48.787904 20535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/master-0/data/instance:
uuid: "ddf3dabc34e845808e15dde382d9a103"
format_stamp: "Formatted at 2026-09-25 15:13:48 on dist-test-slave-p468"
server_key: "64391272b1f00de51932f12f7646bc2d"
server_key_iv: "df90e05d8c2009d42a3dc7271b9c1156"
server_key_version: "encryptionkey@0"
I20260925 15:13:48.788632 20535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/master-0/wal/instance:
uuid: "ddf3dabc34e845808e15dde382d9a103"
format_stamp: "Formatted at 2026-09-25 15:13:48 on dist-test-slave-p468"
server_key: "64391272b1f00de51932f12f7646bc2d"
server_key_iv: "df90e05d8c2009d42a3dc7271b9c1156"
server_key_version: "encryptionkey@0"
I20260925 15:13:48.793537 20535 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 15:13:48.796864 20550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:48.798090 20535 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 15:13:48.798280 20535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/master-0/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/master-0/wal
uuid: "ddf3dabc34e845808e15dde382d9a103"
format_stamp: "Formatted at 2026-09-25 15:13:48 on dist-test-slave-p468"
server_key: "64391272b1f00de51932f12f7646bc2d"
server_key_iv: "df90e05d8c2009d42a3dc7271b9c1156"
server_key_version: "encryptionkey@0"
I20260925 15:13:48.798472 20535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:13:48.828562 20535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:13:48.832543 20535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:13:48.832831 20535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:13:48.850659 20535 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.62:43599
I20260925 15:13:48.850693 20601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.62:43599 every 8 connection(s)
I20260925 15:13:48.851975 20535 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
I20260925 15:13:48.855019 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 20535
I20260925 15:13:48.855212 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/master-0/wal/instance
I20260925 15:13:48.855672 20602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:13:48.855862 12320 external_mini_cluster.cc:1428] Setting key 4e1338589bda27cf3318db055c6c9607
I20260925 15:13:48.867480 20602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddf3dabc34e845808e15dde382d9a103: Bootstrap starting.
Sep 25 15:13:48 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349228, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
I20260925 15:13:48.870945 20602 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ddf3dabc34e845808e15dde382d9a103: Neither blocks nor log segments found. Creating new log.
I20260925 15:13:48.872015 20602 log.cc:824] T 00000000000000000000000000000000 P ddf3dabc34e845808e15dde382d9a103: Log is configured to *not* fsync() on all Append() calls
I20260925 15:13:48.875072 20602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddf3dabc34e845808e15dde382d9a103: No bootstrap required, opened a new log
I20260925 15:13:48.877792 20605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:48.857773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34096 (local address 127.12.8.62:43599)
0925 15:13:48.858638 (+ 865us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:48.858649 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:48.858693 (+ 44us) server_negotiation.cc:407] Connection header received
0925 15:13:48.860392 (+ 1699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:48.860411 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:48.860781 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:48.861155 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:48.862687 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:48.863922 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:48.865136 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:48.865817 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:48.866009 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:48.870286 (+ 4277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:48.870294 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:48.870306 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:48.870343 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:48.873827 (+ 3484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:48.874631 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:48.874637 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:48.874642 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:48.874800 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:48.875220 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:48.875225 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:48.875229 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:48.875776 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:48.876011 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:48.876426 (+ 415us) server_negotiation.cc:299] Negotiation successful
0925 15:13:48.876772 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":674,"thread_start_us":406,"threads_started":1}
I20260925 15:13:48.882925 20602 raft_consensus.cc:348] T 00000000000000000000000000000000 P ddf3dabc34e845808e15dde382d9a103 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddf3dabc34e845808e15dde382d9a103" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 43599 } }
I20260925 15:13:48.883417 20602 raft_consensus.cc:374] T 00000000000000000000000000000000 P ddf3dabc34e845808e15dde382d9a103 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:13:48.883527 20602 raft_consensus.cc:729] T 00000000000000000000000000000000 P ddf3dabc34e845808e15dde382d9a103 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddf3dabc34e845808e15dde382d9a103, State: Initialized, Role: FOLLOWER
I20260925 15:13:48.884056 20602 consensus_queue.cc:260] T 00000000000000000000000000000000 P ddf3dabc34e845808e15dde382d9a103 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddf3dabc34e845808e15dde382d9a103" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 43599 } }
I20260925 15:13:48.884254 20602 raft_consensus.cc:388] T 00000000000000000000000000000000 P ddf3dabc34e845808e15dde382d9a103 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:13:48.884348 20602 raft_consensus.cc:482] T 00000000000000000000000000000000 P ddf3dabc34e845808e15dde382d9a103 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:13:48.884455 20602 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ddf3dabc34e845808e15dde382d9a103 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:13:48.886574 20602 raft_consensus.cc:504] T 00000000000000000000000000000000 P ddf3dabc34e845808e15dde382d9a103 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddf3dabc34e845808e15dde382d9a103" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 43599 } }
I20260925 15:13:48.886992 20602 leader_election.cc:302] T 00000000000000000000000000000000 P ddf3dabc34e845808e15dde382d9a103 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ddf3dabc34e845808e15dde382d9a103; no voters:
I20260925 15:13:48.887511 20602 leader_election.cc:288] T 00000000000000000000000000000000 P ddf3dabc34e845808e15dde382d9a103 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:13:48.887612 20607 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ddf3dabc34e845808e15dde382d9a103 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:13:48.888288 20607 raft_consensus.cc:686] T 00000000000000000000000000000000 P ddf3dabc34e845808e15dde382d9a103 [term 1 LEADER]: Becoming Leader. State: Replica: ddf3dabc34e845808e15dde382d9a103, State: Running, Role: LEADER
I20260925 15:13:48.888758 20607 consensus_queue.cc:237] T 00000000000000000000000000000000 P ddf3dabc34e845808e15dde382d9a103 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddf3dabc34e845808e15dde382d9a103" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 43599 } }
I20260925 15:13:48.889423 20602 sys_catalog.cc:564] T 00000000000000000000000000000000 P ddf3dabc34e845808e15dde382d9a103 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:13:48.891913 20609 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddf3dabc34e845808e15dde382d9a103 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ddf3dabc34e845808e15dde382d9a103. Latest consensus state: current_term: 1 leader_uuid: "ddf3dabc34e845808e15dde382d9a103" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddf3dabc34e845808e15dde382d9a103" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 43599 } } }
I20260925 15:13:48.892330 20609 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddf3dabc34e845808e15dde382d9a103 [sys.catalog]: This master's current role is: LEADER
I20260925 15:13:48.892930 20608 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddf3dabc34e845808e15dde382d9a103 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ddf3dabc34e845808e15dde382d9a103" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddf3dabc34e845808e15dde382d9a103" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 43599 } } }
I20260925 15:13:48.893209 20608 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddf3dabc34e845808e15dde382d9a103 [sys.catalog]: This master's current role is: LEADER
I20260925 15:13:48.893471 20614 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:13:48.899833 20614 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:13:48.909718 20614 catalog_manager.cc:1347] Generated new cluster ID: 32cfe71ac85d45028ebced3b90a3d4e8
I20260925 15:13:48.909844 20614 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:13:48.943491 20614 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:13:48.944913 20614 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:13:48.955744 20614 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ddf3dabc34e845808e15dde382d9a103: Generated new TSK 0
I20260925 15:13:48.956529 20614 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.8.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.1.keytab.
Entry for principal kudu/127.12.8.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.1.keytab.
I20260925 15:13:49.015671 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.1:0
--local_ip_for_outbound_sockets=127.12.8.1
--webserver_interface=127.12.8.1
--webserver_port=0
--tserver_master_addrs=127.12.8.62:43599
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.1.keytab
--principal=kudu/127.12.8.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:39047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:13:49.353487 20628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:13:49.353756 20628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:13:49.353904 20628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:13:49.361023 20628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:13:49.361122 20628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:13:49.361322 20628 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.1
I20260925 15:13:49.370731 20628 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:39047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.1.keytab
--principal=kudu/127.12.8.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.8.1
--webserver_port=0
--tserver_master_addrs=127.12.8.62:43599
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.1
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:13:49.372018 20628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:13:49.373296 20628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:13:49.386036 20634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:49.386096 20636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:49.386524 20633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:13:49.386837 20628 server_base.cc:1034] running on GCE node
I20260925 15:13:49.387387 20628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:13:49.388274 20628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:13:49.389490 20628 hybrid_clock.cc:648] HybridClock initialized: now 1790349229389457 us; error 46 us; skew 500 ppm
I20260925 15:13:49.389804 20628 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:13:49 dist-test-slave-p468 krb5kdc[20522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349229, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:13:49.392920 20628 init.cc:377] Logged in from keytab as kudu/127.12.8.1@KRBTEST.COM (short username kudu)
I20260925 15:13:49.394528 20628 webserver.cc:460] Webserver started at http://127.12.8.1:43283/ using document root <none> and password file <none>
I20260925 15:13:49.395200 20628 fs_manager.cc:359] Metadata directory not provided
I20260925 15:13:49.395340 20628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:13:49.395637 20628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:13:49.398476 20628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/ts-0/data/instance:
uuid: "afd286ba6ffc4fb280f886b0d551bbac"
format_stamp: "Formatted at 2026-09-25 15:13:49 on dist-test-slave-p468"
server_key: "ce6b986ad523df4291504f0bd0df9d02"
server_key_iv: "84d984bfdd97845a7e72804fc27bf1a4"
server_key_version: "encryptionkey@0"
I20260925 15:13:49.399225 20628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/ts-0/wal/instance:
uuid: "afd286ba6ffc4fb280f886b0d551bbac"
format_stamp: "Formatted at 2026-09-25 15:13:49 on dist-test-slave-p468"
server_key: "ce6b986ad523df4291504f0bd0df9d02"
server_key_iv: "84d984bfdd97845a7e72804fc27bf1a4"
server_key_version: "encryptionkey@0"
I20260925 15:13:49.404247 20628 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 15:13:49.407529 20643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:49.408896 20628 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 15:13:49.409169 20628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/ts-0/wal
uuid: "afd286ba6ffc4fb280f886b0d551bbac"
format_stamp: "Formatted at 2026-09-25 15:13:49 on dist-test-slave-p468"
server_key: "ce6b986ad523df4291504f0bd0df9d02"
server_key_iv: "84d984bfdd97845a7e72804fc27bf1a4"
server_key_version: "encryptionkey@0"
I20260925 15:13:49.409376 20628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:13:49.424420 20628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:13:49.428504 20628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:13:49.428781 20628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:13:49.429663 20628 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:13:49.430956 20628 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:13:49.431054 20628 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:49.431200 20628 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:13:49.431279 20628 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:49.463635 20628 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.1:42053
I20260925 15:13:49.463680 20755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.1:42053 every 8 connection(s)
I20260925 15:13:49.464890 20628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/ts-0/data/info.pb
I20260925 15:13:49.468461 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 20628
I20260925 15:13:49.468662 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/ts-0/wal/instance
I20260925 15:13:49.469338 12320 external_mini_cluster.cc:1428] Setting key e441b240ff09f568bb7a6521faf5b728
Sep 25 15:13:49 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349229, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.1@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
I20260925 15:13:49.485713 20758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:49.467401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.1:45941 (local address 127.12.8.62:43599)
0925 15:13:49.467786 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:49.467791 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:49.468987 (+ 1196us) server_negotiation.cc:407] Connection header received
0925 15:13:49.470883 (+ 1896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:49.470889 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:49.470970 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:49.471095 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:49.472784 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:49.473495 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:49.475181 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:49.475575 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:49.475710 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:49.479699 (+ 3989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:49.479704 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:49.479708 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:49.479758 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:49.482507 (+ 2749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:49.483448 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:49.483452 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:49.483454 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:49.483566 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:49.484051 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:49.484054 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:49.484056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:49.484303 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:49.484408 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:49.485316 (+ 908us) server_negotiation.cc:299] Negotiation successful
0925 15:13:49.485476 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":193,"threads_started":1}
I20260925 15:13:49.487408 20759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:49.467979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:43599 (local address 127.12.8.1:45941)
0925 15:13:49.468811 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:49.468861 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:49.470624 (+ 1763us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:49.471342 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:49.471350 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:49.471901 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:13:49.472610 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:49.472628 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:49.473689 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:49.473694 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:49.475025 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:49.475038 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:49.475800 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:49.475809 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:49.476051 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:49.476734 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:13:49.476759 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:13:49.479473 (+ 2714us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:13:49.482648 (+ 3175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:13:49.482661 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:13:49.482681 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:13:49.483330 (+ 649us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:13:49.483671 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:13:49.483678 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:13:49.483684 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:13:49.483939 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:13:49.484423 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:13:49.484434 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:13:49.484923 (+ 489us) client_negotiation.cc:769] Sending connection context
0925 15:13:49.485237 (+ 314us) client_negotiation.cc:240] Negotiation successful
0925 15:13:49.485834 (+ 597us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":350,"threads_started":1}
I20260925 15:13:49.489051 20756 heartbeater.cc:347] Connected to a master server at 127.12.8.62:43599
I20260925 15:13:49.489344 20756 heartbeater.cc:464] Registering TS with master...
I20260925 15:13:49.490132 20756 heartbeater.cc:511] Master 127.12.8.62:43599 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.12.8.2@KRBTEST.COM; defaulting to no policy
I20260925 15:13:49.492339 20567 ts_manager.cc:194] Registered new tserver with Master: afd286ba6ffc4fb280f886b0d551bbac (127.12.8.1:42053)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.2@KRBTEST.COM" created.
I20260925 15:13:49.493978 20567 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.1@KRBTEST.COM'} at 127.12.8.1:45941
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.2.keytab.
Entry for principal kudu/127.12.8.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.2.keytab.
I20260925 15:13:49.519408 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.2:0
--local_ip_for_outbound_sockets=127.12.8.2
--webserver_interface=127.12.8.2
--webserver_port=0
--tserver_master_addrs=127.12.8.62:43599
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.2.keytab
--principal=kudu/127.12.8.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:39047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:13:49.835915 20762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:13:49.836181 20762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:13:49.836404 20762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:13:49.844010 20762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:13:49.844142 20762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:13:49.844321 20762 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.2
I20260925 15:13:49.853632 20762 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:39047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.2.keytab
--principal=kudu/127.12.8.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.8.2
--webserver_port=0
--tserver_master_addrs=127.12.8.62:43599
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.2
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:13:49.854840 20762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:13:49.856046 20762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:13:49.871510 20767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:49.871647 20770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:49.871683 20768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:13:49.872157 20762 server_base.cc:1034] running on GCE node
I20260925 15:13:49.872687 20762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:13:49.873584 20762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:13:49.874773 20762 hybrid_clock.cc:648] HybridClock initialized: now 1790349229874740 us; error 37 us; skew 500 ppm
I20260925 15:13:49.875036 20762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:13:49 dist-test-slave-p468 krb5kdc[20522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349229, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:13:49.878019 20762 init.cc:377] Logged in from keytab as kudu/127.12.8.2@KRBTEST.COM (short username kudu)
I20260925 15:13:49.879355 20762 webserver.cc:460] Webserver started at http://127.12.8.2:39107/ using document root <none> and password file <none>
I20260925 15:13:49.879969 20762 fs_manager.cc:359] Metadata directory not provided
I20260925 15:13:49.880064 20762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:13:49.880332 20762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:13:49.883159 20762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/ts-1/data/instance:
uuid: "d1b34478919649f6ba96e8afd4868274"
format_stamp: "Formatted at 2026-09-25 15:13:49 on dist-test-slave-p468"
server_key: "5052bccf88eb18c5f8a2649b43fa2de4"
server_key_iv: "52ab1d6d8057856db09520dc247663f8"
server_key_version: "encryptionkey@0"
I20260925 15:13:49.883890 20762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/ts-1/wal/instance:
uuid: "d1b34478919649f6ba96e8afd4868274"
format_stamp: "Formatted at 2026-09-25 15:13:49 on dist-test-slave-p468"
server_key: "5052bccf88eb18c5f8a2649b43fa2de4"
server_key_iv: "52ab1d6d8057856db09520dc247663f8"
server_key_version: "encryptionkey@0"
I20260925 15:13:49.888826 20762 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 15:13:49.892273 20777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:49.893528 20762 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 15:13:49.893656 20762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/ts-1/wal
uuid: "d1b34478919649f6ba96e8afd4868274"
format_stamp: "Formatted at 2026-09-25 15:13:49 on dist-test-slave-p468"
server_key: "5052bccf88eb18c5f8a2649b43fa2de4"
server_key_iv: "52ab1d6d8057856db09520dc247663f8"
server_key_version: "encryptionkey@0"
I20260925 15:13:49.893818 20762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:13:49.919509 20762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:13:49.923779 20762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:13:49.924009 20762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:13:49.924854 20762 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:13:49.926064 20762 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:13:49.926134 20762 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:49.926280 20762 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:13:49.926331 20762 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:49.957609 20762 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.2:41741
I20260925 15:13:49.957655 20889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.2:41741 every 8 connection(s)
I20260925 15:13:49.958858 20762 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/ts-1/data/info.pb
I20260925 15:13:49.968531 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 20762
I20260925 15:13:49.968745 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/ts-1/wal/instance
I20260925 15:13:49.969277 12320 external_mini_cluster.cc:1428] Setting key 7a7896e5a2c132efd2884eb169d007ce
Sep 25 15:13:49 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349229, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.2@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
I20260925 15:13:49.977778 20758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:49.961333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.2:36831 (local address 127.12.8.62:43599)
0925 15:13:49.961497 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:49.961501 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:49.962667 (+ 1166us) server_negotiation.cc:407] Connection header received
0925 15:13:49.964379 (+ 1712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:49.964383 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:49.964449 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:49.964536 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:49.965913 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:49.966688 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:49.968128 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:49.968533 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:49.968626 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:49.972322 (+ 3696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:49.972328 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:49.972331 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:49.972360 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:49.974943 (+ 2583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:49.975626 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:49.975633 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:49.975637 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:49.975760 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:49.976153 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:49.976160 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:49.976166 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:49.976421 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:49.976569 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:49.977213 (+ 644us) server_negotiation.cc:299] Negotiation successful
0925 15:13:49.977430 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 15:13:49.978718 20892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:49.961789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:43599 (local address 127.12.8.2:36831)
0925 15:13:49.962509 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:49.962552 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:49.964147 (+ 1595us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:49.964725 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:49.964733 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:49.965247 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:13:49.965774 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:49.965790 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:49.966818 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:49.966822 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:49.968000 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:49.968015 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:49.968756 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:49.968766 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:49.969043 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:49.969734 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:13:49.969762 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:13:49.972117 (+ 2355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:13:49.975079 (+ 2962us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:13:49.975087 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:13:49.975103 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:13:49.975508 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:13:49.975865 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:13:49.975871 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:13:49.975875 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:13:49.976048 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:13:49.976541 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:13:49.976548 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:13:49.976875 (+ 327us) client_negotiation.cc:769] Sending connection context
0925 15:13:49.977132 (+ 257us) client_negotiation.cc:240] Negotiation successful
0925 15:13:49.977621 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":336,"threads_started":1}
I20260925 15:13:49.980167 20890 heartbeater.cc:347] Connected to a master server at 127.12.8.62:43599
I20260925 15:13:49.980459 20890 heartbeater.cc:464] Registering TS with master...
I20260925 15:13:49.981223 20890 heartbeater.cc:511] Master 127.12.8.62:43599 requested a full tablet report, sending...
I20260925 15:13:49.982700 20567 ts_manager.cc:194] Registered new tserver with Master: d1b34478919649f6ba96e8afd4868274 (127.12.8.2:41741)
I20260925 15:13:49.983933 20567 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.2@KRBTEST.COM'} at 127.12.8.2:36831
WARNING: no policy specified for kudu/127.12.8.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.3.keytab.
Entry for principal kudu/127.12.8.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.3.keytab.
I20260925 15:13:50.017699 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.3:0
--local_ip_for_outbound_sockets=127.12.8.3
--webserver_interface=127.12.8.3
--webserver_port=0
--tserver_master_addrs=127.12.8.62:43599
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.3.keytab
--principal=kudu/127.12.8.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:39047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:13:50.336138 20895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:13:50.336393 20895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:13:50.336485 20895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:13:50.343439 20895 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:13:50.343542 20895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:13:50.343652 20895 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.3
I20260925 15:13:50.352885 20895 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:39047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.3.keytab
--principal=kudu/127.12.8.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.8.3
--webserver_port=0
--tserver_master_addrs=127.12.8.62:43599
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.3
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:13:50.354080 20895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:13:50.355360 20895 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:13:50.367815 20901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:50.367898 20900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:50.367875 20903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:13:50.368963 20895 server_base.cc:1034] running on GCE node
I20260925 15:13:50.369550 20895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:13:50.370494 20895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:13:50.371706 20895 hybrid_clock.cc:648] HybridClock initialized: now 1790349230371662 us; error 56 us; skew 500 ppm
I20260925 15:13:50.372000 20895 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:13:50 dist-test-slave-p468 krb5kdc[20522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349230, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:13:50.375095 20895 init.cc:377] Logged in from keytab as kudu/127.12.8.3@KRBTEST.COM (short username kudu)
I20260925 15:13:50.376361 20895 webserver.cc:460] Webserver started at http://127.12.8.3:46249/ using document root <none> and password file <none>
I20260925 15:13:50.376999 20895 fs_manager.cc:359] Metadata directory not provided
I20260925 15:13:50.377137 20895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:13:50.377434 20895 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:13:50.380204 20895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/ts-2/data/instance:
uuid: "730ff31af6fe48f69c09563dba091568"
format_stamp: "Formatted at 2026-09-25 15:13:50 on dist-test-slave-p468"
server_key: "f7edfc5ab03104a29044b1e811eab48b"
server_key_iv: "7d8a2bff041e638c094d61efd8bfd2f4"
server_key_version: "encryptionkey@0"
I20260925 15:13:50.380959 20895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/ts-2/wal/instance:
uuid: "730ff31af6fe48f69c09563dba091568"
format_stamp: "Formatted at 2026-09-25 15:13:50 on dist-test-slave-p468"
server_key: "f7edfc5ab03104a29044b1e811eab48b"
server_key_iv: "7d8a2bff041e638c094d61efd8bfd2f4"
server_key_version: "encryptionkey@0"
I20260925 15:13:50.385905 20895 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260925 15:13:50.389137 20910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:50.390460 20895 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:13:50.390615 20895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/ts-2/wal
uuid: "730ff31af6fe48f69c09563dba091568"
format_stamp: "Formatted at 2026-09-25 15:13:50 on dist-test-slave-p468"
server_key: "f7edfc5ab03104a29044b1e811eab48b"
server_key_iv: "7d8a2bff041e638c094d61efd8bfd2f4"
server_key_version: "encryptionkey@0"
I20260925 15:13:50.390800 20895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:13:50.409577 20895 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:13:50.413303 20895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:13:50.413570 20895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:13:50.414558 20895 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:13:50.415860 20895 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:13:50.415961 20895 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:50.416093 20895 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:13:50.416162 20895 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:50.448004 20895 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.3:34059
I20260925 15:13:50.448046 21022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.3:34059 every 8 connection(s)
I20260925 15:13:50.449247 20895 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/ts-2/data/info.pb
I20260925 15:13:50.457376 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 20895
I20260925 15:13:50.457535 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/ts-2/wal/instance
I20260925 15:13:50.457959 12320 external_mini_cluster.cc:1428] Setting key ddc7d6709a1b2e88ba6e9bc23bc09ea1
Sep 25 15:13:50 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349230, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.3@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
I20260925 15:13:50.469794 20758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:50.451664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.3:40139 (local address 127.12.8.62:43599)
0925 15:13:50.451858 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:50.451862 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:50.452944 (+ 1082us) server_negotiation.cc:407] Connection header received
0925 15:13:50.454774 (+ 1830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:50.454780 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:50.454862 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:50.454973 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:50.456404 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:50.457151 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:50.459057 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:50.459710 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:50.459810 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:50.463494 (+ 3684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:50.463501 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:50.463505 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:50.463543 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:50.466891 (+ 3348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:50.467686 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:50.467692 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:50.467696 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:50.467838 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:50.468278 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:50.468282 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:50.468285 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:50.468622 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:50.468771 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:50.469383 (+ 612us) server_negotiation.cc:299] Negotiation successful
0925 15:13:50.469557 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 15:13:50.470762 21025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:50.452117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:43599 (local address 127.12.8.3:40139)
0925 15:13:50.452794 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:50.452831 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:50.454490 (+ 1659us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:50.455206 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:50.455214 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:50.455695 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:13:50.456237 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:50.456253 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:50.457299 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:50.457305 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:50.458909 (+ 1604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:50.458926 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:50.459926 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:50.459932 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:50.460198 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:50.460968 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:13:50.461024 (+ 56us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:13:50.463290 (+ 2266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:13:50.467040 (+ 3750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:13:50.467047 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:13:50.467058 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:13:50.467546 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:13:50.467973 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:13:50.467977 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:13:50.467979 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:13:50.468154 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:13:50.468735 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:13:50.468740 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:13:50.469024 (+ 284us) client_negotiation.cc:769] Sending connection context
0925 15:13:50.469246 (+ 222us) client_negotiation.cc:240] Negotiation successful
0925 15:13:50.469599 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":267,"threads_started":1}
I20260925 15:13:50.472227 21023 heartbeater.cc:347] Connected to a master server at 127.12.8.62:43599
I20260925 15:13:50.472556 21023 heartbeater.cc:464] Registering TS with master...
I20260925 15:13:50.473361 21023 heartbeater.cc:511] Master 127.12.8.62:43599 requested a full tablet report, sending...
I20260925 15:13:50.474828 20567 ts_manager.cc:194] Registered new tserver with Master: 730ff31af6fe48f69c09563dba091568 (127.12.8.3:34059)
I20260925 15:13:50.475610 20567 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.3@KRBTEST.COM'} at 127.12.8.3:40139
I20260925 15:13:50.485378 12320 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:13:50.497004 20756 heartbeater.cc:503] Master 127.12.8.62:43599 was elected leader, sending a full tablet report...
WARNING: no policy specified for oryx/127.12.8.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.8.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.8.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/krb5kdc/oryx_127.12.8.1.keytab.
Entry for principal oryx/127.12.8.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/krb5kdc/oryx_127.12.8.1.keytab.
WARNING: no policy specified for oryx/127.12.8.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.8.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.8.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/krb5kdc/oryx_127.12.8.2.keytab.
Entry for principal oryx/127.12.8.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/krb5kdc/oryx_127.12.8.2.keytab.
WARNING: no policy specified for oryx/127.12.8.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.8.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.8.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/krb5kdc/oryx_127.12.8.3.keytab.
Entry for principal oryx/127.12.8.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/krb5kdc/oryx_127.12.8.3.keytab.
I20260925 15:13:50.616933 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 20628
Sep 25 15:13:50 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349228, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.1@KRBTEST.COM
I20260925 15:13:50.632383 21033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:50.618165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34608 (local address 127.12.8.1:42053)
0925 15:13:50.618781 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:50.618788 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:50.618816 (+ 28us) server_negotiation.cc:407] Connection header received
0925 15:13:50.618878 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:50.618885 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:50.619114 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:50.619242 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:50.620532 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:50.621521 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:50.622745 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:50.623295 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:50.623391 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:50.627120 (+ 3729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:50.627131 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:50.627145 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:50.627181 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:50.629940 (+ 2759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:50.630619 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:50.630626 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:50.630628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:50.630752 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:50.631037 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:50.631040 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:50.631042 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:50.631354 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:50.631566 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:50.631795 (+ 229us) server_negotiation.cc:299] Negotiation successful
0925 15:13:50.631983 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":305,"threads_started":1}
I20260925 15:13:50.633592 20751 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:13:50.715946 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 20628
I20260925 15:13:50.731734 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 20762
Sep 25 15:13:50 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349228, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.2@KRBTEST.COM
I20260925 15:13:50.746914 21035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:50.732593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48540 (local address 127.12.8.2:41741)
0925 15:13:50.733161 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:50.733168 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:50.733193 (+ 25us) server_negotiation.cc:407] Connection header received
0925 15:13:50.733259 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:50.733267 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:50.733473 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:50.733602 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:50.734948 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:50.735949 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:50.737244 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:50.737783 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:50.737881 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:50.741863 (+ 3982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:50.741875 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:50.741888 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:50.741919 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:50.744532 (+ 2613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:50.745115 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:50.745122 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:50.745124 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:50.745254 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:50.745541 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:50.745544 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:50.745546 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:50.745849 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:50.746059 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:50.746453 (+ 394us) server_negotiation.cc:299] Negotiation successful
0925 15:13:50.746636 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":298,"threads_started":1}
I20260925 15:13:50.748136 20885 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:13:50.828840 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 20762
I20260925 15:13:50.844213 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 20895
Sep 25 15:13:50 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349228, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.3@KRBTEST.COM
I20260925 15:13:50.861359 21037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:50.845276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37540 (local address 127.12.8.3:34059)
0925 15:13:50.845961 (+ 685us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:50.845968 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:50.845996 (+ 28us) server_negotiation.cc:407] Connection header received
0925 15:13:50.846058 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:50.846064 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:50.846328 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:50.846463 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:50.847702 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:50.848732 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:50.849930 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:50.850625 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:50.850746 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:50.855983 (+ 5237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:50.855993 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:50.856007 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:50.856042 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:50.858842 (+ 2800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:50.859554 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:50.859561 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:50.859563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:50.859711 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:50.860030 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:50.860033 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:50.860035 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:50.860356 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:50.860571 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:50.860804 (+ 233us) server_negotiation.cc:299] Negotiation successful
0925 15:13:50.860986 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":393,"threads_started":1}
I20260925 15:13:50.862740 21018 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:13:50.943965 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 20895
I20260925 15:13:50.959218 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 20535
I20260925 15:13:50.960470 20597 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:13:51.062171 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 20535
I20260925 15:13:51.082646 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.62:43599
--webserver_interface=127.12.8.62
--webserver_port=42379
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab
--principal=kudu/127.12.8.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:39047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.62:43599
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:13:51.403919 21040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:13:51.404191 21040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:13:51.404340 21040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:13:51.411304 21040 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:13:51.411408 21040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:13:51.411491 21040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:13:51.411593 21040 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:13:51.411652 21040 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:13:51.421060 21040 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:39047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.8.62:43599
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab
--principal=kudu/127.12.8.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.62:43599
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.8.62
--webserver_port=42379
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:13:51.422508 21040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:13:51.423883 21040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:13:51.435128 21045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:51.435164 21048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:51.435375 21046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:13:51.436386 21040 server_base.cc:1034] running on GCE node
I20260925 15:13:51.437157 21040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:13:51.438638 21040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:13:51.439863 21040 hybrid_clock.cc:648] HybridClock initialized: now 1790349231439821 us; error 53 us; skew 500 ppm
I20260925 15:13:51.440189 21040 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:13:51 dist-test-slave-p468 krb5kdc[20522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349231, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:13:51.443240 21040 init.cc:377] Logged in from keytab as kudu/127.12.8.62@KRBTEST.COM (short username kudu)
I20260925 15:13:51.444670 21040 webserver.cc:460] Webserver started at http://127.12.8.62:42379/ using document root <none> and password file <none>
I20260925 15:13:51.445273 21040 fs_manager.cc:359] Metadata directory not provided
I20260925 15:13:51.445380 21040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:13:51.450582 21040 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:13:51.453214 21055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:51.454577 21040 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:13:51.454772 21040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/master-0/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/master-0/wal
uuid: "ddf3dabc34e845808e15dde382d9a103"
format_stamp: "Formatted at 2026-09-25 15:13:48 on dist-test-slave-p468"
server_key: "64391272b1f00de51932f12f7646bc2d"
server_key_iv: "df90e05d8c2009d42a3dc7271b9c1156"
server_key_version: "encryptionkey@0"
I20260925 15:13:51.455715 21040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:13:51.469033 21040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:13:51.472926 21040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:13:51.473182 21040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:13:51.491570 21040 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.62:43599
I20260925 15:13:51.491590 21106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.62:43599 every 8 connection(s)
I20260925 15:13:51.493068 21040 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
I20260925 15:13:51.499743 21107 sys_catalog.cc:263] Verifying existing consensus state
I20260925 15:13:51.502413 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 21040
I20260925 15:13:51.502935 21107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddf3dabc34e845808e15dde382d9a103: Bootstrap starting.
I20260925 15:13:51.503227 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.1:42053
--local_ip_for_outbound_sockets=127.12.8.1
--tserver_master_addrs=127.12.8.62:43599
--webserver_port=43283
--webserver_interface=127.12.8.1
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.1.keytab
--principal=kudu/127.12.8.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:39047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.12.8.1
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/krb5kdc/oryx_127.12.8.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:13:51.521088 21107 log.cc:824] T 00000000000000000000000000000000 P ddf3dabc34e845808e15dde382d9a103: Log is configured to *not* fsync() on all Append() calls
I20260925 15:13:51.529484 21107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddf3dabc34e845808e15dde382d9a103: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 15:13:51.529992 21107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddf3dabc34e845808e15dde382d9a103: Bootstrap complete.
I20260925 15:13:51.538836 21107 raft_consensus.cc:348] T 00000000000000000000000000000000 P ddf3dabc34e845808e15dde382d9a103 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddf3dabc34e845808e15dde382d9a103" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 43599 } }
I20260925 15:13:51.539626 21107 raft_consensus.cc:729] T 00000000000000000000000000000000 P ddf3dabc34e845808e15dde382d9a103 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddf3dabc34e845808e15dde382d9a103, State: Initialized, Role: FOLLOWER
I20260925 15:13:51.540280 21107 consensus_queue.cc:260] T 00000000000000000000000000000000 P ddf3dabc34e845808e15dde382d9a103 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddf3dabc34e845808e15dde382d9a103" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 43599 } }
I20260925 15:13:51.540501 21107 raft_consensus.cc:388] T 00000000000000000000000000000000 P ddf3dabc34e845808e15dde382d9a103 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:13:51.540611 21107 raft_consensus.cc:482] T 00000000000000000000000000000000 P ddf3dabc34e845808e15dde382d9a103 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:13:51.540791 21107 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ddf3dabc34e845808e15dde382d9a103 [term 1 FOLLOWER]: Advancing to term 2
I20260925 15:13:51.544366 21107 raft_consensus.cc:504] T 00000000000000000000000000000000 P ddf3dabc34e845808e15dde382d9a103 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddf3dabc34e845808e15dde382d9a103" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 43599 } }
I20260925 15:13:51.544828 21107 leader_election.cc:302] T 00000000000000000000000000000000 P ddf3dabc34e845808e15dde382d9a103 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ddf3dabc34e845808e15dde382d9a103; no voters:
I20260925 15:13:51.545460 21107 leader_election.cc:288] T 00000000000000000000000000000000 P ddf3dabc34e845808e15dde382d9a103 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 15:13:51.545588 21111 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ddf3dabc34e845808e15dde382d9a103 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 15:13:51.546875 21111 raft_consensus.cc:686] T 00000000000000000000000000000000 P ddf3dabc34e845808e15dde382d9a103 [term 2 LEADER]: Becoming Leader. State: Replica: ddf3dabc34e845808e15dde382d9a103, State: Running, Role: LEADER
I20260925 15:13:51.547472 21111 consensus_queue.cc:237] T 00000000000000000000000000000000 P ddf3dabc34e845808e15dde382d9a103 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddf3dabc34e845808e15dde382d9a103" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 43599 } }
I20260925 15:13:51.547660 21107 sys_catalog.cc:564] T 00000000000000000000000000000000 P ddf3dabc34e845808e15dde382d9a103 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:13:51.549988 21113 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddf3dabc34e845808e15dde382d9a103 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ddf3dabc34e845808e15dde382d9a103. Latest consensus state: current_term: 2 leader_uuid: "ddf3dabc34e845808e15dde382d9a103" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddf3dabc34e845808e15dde382d9a103" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 43599 } } }
I20260925 15:13:51.550436 21113 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddf3dabc34e845808e15dde382d9a103 [sys.catalog]: This master's current role is: LEADER
I20260925 15:13:51.551007 21112 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddf3dabc34e845808e15dde382d9a103 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ddf3dabc34e845808e15dde382d9a103" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddf3dabc34e845808e15dde382d9a103" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 43599 } } }
I20260925 15:13:51.551365 21112 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddf3dabc34e845808e15dde382d9a103 [sys.catalog]: This master's current role is: LEADER
I20260925 15:13:51.556308 21118 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:13:51.562084 21118 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:13:51.563784 21118 catalog_manager.cc:1259] Loaded cluster ID: 32cfe71ac85d45028ebced3b90a3d4e8
I20260925 15:13:51.563961 21118 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:13:51.568698 21128 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 15:13:51.568895 21118 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:13:51.570384 21118 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ddf3dabc34e845808e15dde382d9a103: Loaded TSK: 0
I20260925 15:13:51.571417 21118 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 15:13:51.841763 21109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:13:51.842057 21109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:13:51.842269 21109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:13:51.849696 21109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:13:51.849835 21109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:13:51.849996 21109 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.1
I20260925 15:13:51.862720 21109 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:39047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/krb5kdc/oryx_127.12.8.1.keytab
--principal=oryx/127.12.8.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.1:42053
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.8.1
--webserver_port=43283
--tserver_master_addrs=127.12.8.62:43599
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.1
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:13:51.865319 21109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:13:51.867102 21109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:13:51.881619 21134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:51.882012 21136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:51.882381 21133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:13:51.883209 21109 server_base.cc:1034] running on GCE node
I20260925 15:13:51.883932 21109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:13:51.885092 21109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:13:51.886297 21109 hybrid_clock.cc:648] HybridClock initialized: now 1790349231886228 us; error 58 us; skew 500 ppm
I20260925 15:13:51.886674 21109 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:13:51 dist-test-slave-p468 krb5kdc[20522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349231, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.8.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:13:51.890838 21109 init.cc:377] Logged in from keytab as oryx/127.12.8.1@KRBTEST.COM (short username oryx)
I20260925 15:13:51.892628 21109 webserver.cc:460] Webserver started at http://127.12.8.1:43283/ using document root <none> and password file <none>
I20260925 15:13:51.893589 21109 fs_manager.cc:359] Metadata directory not provided
I20260925 15:13:51.893746 21109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:13:51.901031 21109 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 15:13:51.904304 21143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:51.905897 21109 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 15:13:51.906075 21109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/ts-0/wal
uuid: "afd286ba6ffc4fb280f886b0d551bbac"
format_stamp: "Formatted at 2026-09-25 15:13:49 on dist-test-slave-p468"
server_key: "ce6b986ad523df4291504f0bd0df9d02"
server_key_iv: "84d984bfdd97845a7e72804fc27bf1a4"
server_key_version: "encryptionkey@0"
I20260925 15:13:51.907461 21109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:13:51.929409 21109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:13:51.934760 21109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:13:51.935034 21109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:13:51.936215 21109 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:13:51.938005 21109 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:13:51.938115 21109 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:51.938238 21109 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:13:51.938315 21109 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:51.974943 21109 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.1:42053
I20260925 15:13:51.974997 21255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.1:42053 every 8 connection(s)
I20260925 15:13:51.976739 21109 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/ts-0/data/info.pb
I20260925 15:13:51.983767 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 21109
I20260925 15:13:51.984383 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.2:41741
--local_ip_for_outbound_sockets=127.12.8.2
--tserver_master_addrs=127.12.8.62:43599
--webserver_port=39107
--webserver_interface=127.12.8.2
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.2.keytab
--principal=kudu/127.12.8.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:39047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.12.8.2
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/krb5kdc/oryx_127.12.8.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 15:13:51 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.1@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:13:51 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.1@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 15:13:51.995592 21258 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:13:51.979709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.1:47491 (local address 127.12.8.62:43599)
0925 15:13:51.980451 (+ 742us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:51.980468 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:51.980852 (+ 384us) server_negotiation.cc:407] Connection header received
0925 15:13:51.982978 (+ 2126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:51.982991 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:51.983289 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:51.983647 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:51.985215 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:51.986471 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:51.987981 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:51.988676 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:51.989012 (+ 336us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:51.994775 (+ 5763us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.1:47491: BlockingRecv error: recv got EOF from 127.12.8.1:47491 (error 108)
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":241,"threads_started":1}
I20260925 15:13:51.995645 21259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:51.979902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:43599 (local address 127.12.8.1:47491)
0925 15:13:51.980688 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:51.980730 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:51.982431 (+ 1701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:51.983785 (+ 1354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:51.983797 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:51.984387 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:13:51.985010 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:51.985028 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:51.986634 (+ 1606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:51.986638 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:51.987840 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:51.987851 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:51.989030 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:51.989037 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:51.989323 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:51.990022 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:13:51.990046 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:13:51.994380 (+ 4334us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.62:43599: Server oryx/127.12.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":337,"threads_started":1}
W20260925 15:13:51.996369 21256 heartbeater.cc:644] Failed to heartbeat to 127.12.8.62:43599 (0 consecutive failures): Not authorized: Failed to ping master at 127.12.8.62:43599: Client connection negotiation failed: client connection to 127.12.8.62:43599: Server oryx/127.12.8.62@KRBTEST.COM not found in Kerberos database
Sep 25 15:13:51 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.1@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:13:51 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.1@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:13:52.005650 21259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:51.996987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:43599 (local address 127.12.8.1:53171)
0925 15:13:51.997172 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:51.997191 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:51.997301 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:51.997722 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:51.997725 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:51.998015 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:13:51.998154 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:51.998161 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:51.999770 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:51.999775 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:52.000756 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:52.000768 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:52.001860 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:52.001865 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:52.001997 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:52.002616 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:13:52.002633 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:13:52.005458 (+ 2825us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.62:43599: Server oryx/127.12.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":57}
W20260925 15:13:52.006012 21258 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:13:51.997207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.1:53171 (local address 127.12.8.62:43599)
0925 15:13:51.997401 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:51.997409 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:51.997429 (+ 20us) server_negotiation.cc:407] Connection header received
0925 15:13:51.997502 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:51.997510 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:51.997587 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:51.997689 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:51.998322 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:51.999604 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:52.000902 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:52.001688 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:52.001840 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:52.005734 (+ 3894us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.1:53171: BlockingRecv error: recv got EOF from 127.12.8.1:53171 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 25 15:13:52 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.1@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:13:52 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.1@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:13:52.014776 21259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:52.006424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:43599 (local address 127.12.8.1:40031)
0925 15:13:52.006578 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:52.006597 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:52.006708 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:52.007082 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:52.007085 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:52.007335 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:13:52.007467 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:52.007474 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:52.008948 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:52.008952 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:52.009886 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:52.009897 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:52.010833 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:52.010837 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:52.010957 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:52.011585 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:13:52.011602 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:13:52.014574 (+ 2972us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.62:43599: Server oryx/127.12.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":44}
W20260925 15:13:52.015081 21258 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:13:52.006585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.1:40031 (local address 127.12.8.62:43599)
0925 15:13:52.006775 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:52.006783 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:52.006802 (+ 19us) server_negotiation.cc:407] Connection header received
0925 15:13:52.006857 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:52.006864 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:52.006954 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:52.007081 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:52.007598 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:52.008789 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:52.010016 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:52.010694 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:52.010812 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:52.014810 (+ 3998us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.1:40031: BlockingRecv error: recv got EOF from 127.12.8.1:40031 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260925 15:13:52.015470 21256 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 15:13:52.308986 21260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:13:52.309260 21260 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:13:52.309417 21260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:13:52.316793 21260 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:13:52.316916 21260 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:13:52.317088 21260 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.2
I20260925 15:13:52.326658 21260 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:39047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/krb5kdc/oryx_127.12.8.2.keytab
--principal=oryx/127.12.8.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.2:41741
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.8.2
--webserver_port=39107
--tserver_master_addrs=127.12.8.62:43599
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.2
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:13:52.327925 21260 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:13:52.329200 21260 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:13:52.342173 21265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:52.342137 21268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:52.342226 21266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:13:52.343006 21260 server_base.cc:1034] running on GCE node
I20260925 15:13:52.343592 21260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:13:52.344467 21260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:13:52.345693 21260 hybrid_clock.cc:648] HybridClock initialized: now 1790349232345645 us; error 39 us; skew 500 ppm
I20260925 15:13:52.346021 21260 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:13:52 dist-test-slave-p468 krb5kdc[20522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349232, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.8.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:13:52.352223 21260 init.cc:377] Logged in from keytab as oryx/127.12.8.2@KRBTEST.COM (short username oryx)
I20260925 15:13:52.353691 21260 webserver.cc:460] Webserver started at http://127.12.8.2:39107/ using document root <none> and password file <none>
I20260925 15:13:52.354452 21260 fs_manager.cc:359] Metadata directory not provided
I20260925 15:13:52.354584 21260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:13:52.360147 21260 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 15:13:52.362771 21275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:52.364125 21260 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:13:52.364308 21260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/ts-1/wal
uuid: "d1b34478919649f6ba96e8afd4868274"
format_stamp: "Formatted at 2026-09-25 15:13:49 on dist-test-slave-p468"
server_key: "5052bccf88eb18c5f8a2649b43fa2de4"
server_key_iv: "52ab1d6d8057856db09520dc247663f8"
server_key_version: "encryptionkey@0"
I20260925 15:13:52.365248 21260 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:13:52.376677 21260 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:13:52.380693 21260 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:13:52.380965 21260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:13:52.381872 21260 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:13:52.383174 21260 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:13:52.383284 21260 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:52.383405 21260 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:13:52.383496 21260 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:52.414667 21260 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.2:41741
I20260925 15:13:52.414752 21387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.2:41741 every 8 connection(s)
I20260925 15:13:52.416098 21260 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/ts-1/data/info.pb
I20260925 15:13:52.417193 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 21260
I20260925 15:13:52.417949 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.3:34059
--local_ip_for_outbound_sockets=127.12.8.3
--tserver_master_addrs=127.12.8.62:43599
--webserver_port=46249
--webserver_interface=127.12.8.3
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.3.keytab
--principal=kudu/127.12.8.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:39047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.12.8.3
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/krb5kdc/oryx_127.12.8.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 15:13:52 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.2@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:13:52 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.2@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 15:13:52.433949 21258 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:13:52.419024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.2:50927 (local address 127.12.8.62:43599)
0925 15:13:52.419246 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:52.419254 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:52.420463 (+ 1209us) server_negotiation.cc:407] Connection header received
0925 15:13:52.423315 (+ 2852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:52.423319 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:52.423385 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:52.423485 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:52.425412 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:52.426285 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:52.427786 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:52.428191 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:52.428320 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:52.433768 (+ 5448us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.2:50927: BlockingRecv error: recv got EOF from 127.12.8.2:50927 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 15:13:52.435014 21390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:52.419489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:43599 (local address 127.12.8.2:50927)
0925 15:13:52.420274 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:52.420324 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:52.422966 (+ 2642us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:52.423743 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:52.423758 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:52.424513 (+ 755us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:13:52.425237 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:52.425255 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:52.426444 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:52.426448 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:52.427643 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:52.427655 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:52.428404 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:52.428411 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:52.428635 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:52.429283 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:13:52.429303 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:13:52.433628 (+ 4325us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.62:43599: Server oryx/127.12.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":297,"threads_started":1}
W20260925 15:13:52.435685 21388 heartbeater.cc:644] Failed to heartbeat to 127.12.8.62:43599 (0 consecutive failures): Not authorized: Failed to ping master at 127.12.8.62:43599: Client connection negotiation failed: client connection to 127.12.8.62:43599: Server oryx/127.12.8.62@KRBTEST.COM not found in Kerberos database
Sep 25 15:13:52 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.2@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:13:52 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.2@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:13:52.443797 21390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:52.436247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:43599 (local address 127.12.8.2:46893)
0925 15:13:52.436401 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:52.436419 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:52.436517 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:52.436825 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:52.436829 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:52.437151 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:13:52.437288 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:52.437296 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:52.438275 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:52.438281 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:52.439160 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:52.439171 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:52.439860 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:52.439866 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:52.440024 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:52.440661 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:13:52.440682 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:13:52.443493 (+ 2811us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.62:43599: Server oryx/127.12.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 15:13:52.443897 21258 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:13:52.436364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.2:46893 (local address 127.12.8.62:43599)
0925 15:13:52.436535 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:52.436541 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:52.436570 (+ 29us) server_negotiation.cc:407] Connection header received
0925 15:13:52.436639 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:52.436644 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:52.436707 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:52.436804 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:52.437420 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:52.438119 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:52.439296 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:52.439711 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:52.439839 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:52.443676 (+ 3837us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.2:46893: BlockingRecv error: recv got EOF from 127.12.8.2:46893 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 25 15:13:52 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.2@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:13:52 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.2@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:13:52.452905 21390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:52.444464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:43599 (local address 127.12.8.2:42527)
0925 15:13:52.444624 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:52.444643 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:52.444751 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:52.445043 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:52.445048 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:52.445403 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:13:52.445577 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:52.445587 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:52.446580 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:52.446585 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:52.448156 (+ 1571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:52.448170 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:52.448852 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:52.448858 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:52.449040 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:52.449578 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:13:52.449603 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:13:52.452625 (+ 3022us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.62:43599: Server oryx/127.12.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":40}
W20260925 15:13:52.452925 21258 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:13:52.444509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.2:42527 (local address 127.12.8.62:43599)
0925 15:13:52.444647 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:52.444658 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:52.444694 (+ 36us) server_negotiation.cc:407] Connection header received
0925 15:13:52.444862 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:52.444865 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:52.444928 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:52.444998 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:52.445696 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:52.446449 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:52.448281 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:52.448713 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:52.448832 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:52.452777 (+ 3945us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.2:42527: BlockingRecv error: recv got EOF from 127.12.8.2:42527 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260925 15:13:52.453343 21388 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 15:13:52.740167 21391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:13:52.740425 21391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:13:52.740525 21391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:13:52.747501 21391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:13:52.747601 21391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:13:52.747712 21391 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.3
I20260925 15:13:52.757272 21391 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:39047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/krb5kdc/oryx_127.12.8.3.keytab
--principal=oryx/127.12.8.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.3:34059
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.8.3
--webserver_port=46249
--tserver_master_addrs=127.12.8.62:43599
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.3
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:13:52.758502 21391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:13:52.759780 21391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:13:52.772392 21396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:52.772357 21397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:13:52.772442 21399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:13:52.773350 21391 server_base.cc:1034] running on GCE node
I20260925 15:13:52.773926 21391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:13:52.774896 21391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:13:52.776139 21391 hybrid_clock.cc:648] HybridClock initialized: now 1790349232776094 us; error 51 us; skew 500 ppm
I20260925 15:13:52.776461 21391 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:13:52 dist-test-slave-p468 krb5kdc[20522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349232, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.8.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:13:52.779786 21391 init.cc:377] Logged in from keytab as oryx/127.12.8.3@KRBTEST.COM (short username oryx)
I20260925 15:13:52.781229 21391 webserver.cc:460] Webserver started at http://127.12.8.3:46249/ using document root <none> and password file <none>
I20260925 15:13:52.781893 21391 fs_manager.cc:359] Metadata directory not provided
I20260925 15:13:52.782052 21391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:13:52.787345 21391 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 15:13:52.790057 21406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:52.791389 21391 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:13:52.791671 21391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/ts-2/wal
uuid: "730ff31af6fe48f69c09563dba091568"
format_stamp: "Formatted at 2026-09-25 15:13:50 on dist-test-slave-p468"
server_key: "f7edfc5ab03104a29044b1e811eab48b"
server_key_iv: "7d8a2bff041e638c094d61efd8bfd2f4"
server_key_version: "encryptionkey@0"
I20260925 15:13:52.792630 21391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:13:52.815629 21391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:13:52.819572 21391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:13:52.819844 21391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:13:52.820807 21391 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:13:52.822055 21391 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:13:52.822186 21391 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:52.822338 21391 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:13:52.822427 21391 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:13:52.853976 21391 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.3:34059
I20260925 15:13:52.854059 21518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.3:34059 every 8 connection(s)
I20260925 15:13:52.855500 21391 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349162370577-12320-0/minicluster-data/ts-2/data/info.pb
I20260925 15:13:52.861138 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 21391
Sep 25 15:13:52 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.3@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:13:52 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.3@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 15:13:52.873608 21258 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:13:52.858002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.3:40603 (local address 127.12.8.62:43599)
0925 15:13:52.858173 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:52.858178 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:52.859379 (+ 1201us) server_negotiation.cc:407] Connection header received
0925 15:13:52.861499 (+ 2120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:52.861504 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:52.861573 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:52.861666 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:52.863960 (+ 2294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:52.864709 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:52.867022 (+ 2313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:52.867735 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:52.867874 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:52.873426 (+ 5552us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.3:40603: BlockingRecv error: recv got EOF from 127.12.8.3:40603 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 15:13:52.874716 21521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:52.858488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:43599 (local address 127.12.8.3:40603)
0925 15:13:52.859218 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:52.859254 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:52.861169 (+ 1915us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:52.861912 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:52.861924 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:52.862783 (+ 859us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:13:52.863748 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:52.863779 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:52.864857 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:52.864865 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:52.866892 (+ 2027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:52.866904 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:52.867922 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:52.867927 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:52.868132 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:52.868889 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:13:52.868921 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:13:52.873289 (+ 4368us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.62:43599: Server oryx/127.12.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":275,"threads_started":1}
W20260925 15:13:52.875388 21519 heartbeater.cc:644] Failed to heartbeat to 127.12.8.62:43599 (0 consecutive failures): Not authorized: Failed to ping master at 127.12.8.62:43599: Client connection negotiation failed: client connection to 127.12.8.62:43599: Server oryx/127.12.8.62@KRBTEST.COM not found in Kerberos database
I20260925 15:13:52.882313 21523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:52.862256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34110 (local address 127.12.8.62:43599)
0925 15:13:52.866509 (+ 4253us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:52.866514 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:52.866527 (+ 13us) server_negotiation.cc:407] Connection header received
0925 15:13:52.866564 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:52.866569 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:52.866620 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:52.866731 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:52.868702 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:52.869890 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:52.871489 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:52.871966 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:52.872059 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:52.875258 (+ 3199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:13:52.875273 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:13:52.875294 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:13:52.875336 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:13:52.878974 (+ 3638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:52.879711 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:52.879719 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:52.879726 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:52.879984 (+ 258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:13:52.880360 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:13:52.880363 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:13:52.880365 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:13:52.880944 (+ 579us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:13:52.881232 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0925 15:13:52.881631 (+ 399us) server_negotiation.cc:299] Negotiation successful
0925 15:13:52.881974 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":4168,"thread_start_us":4079,"threads_started":1}
Sep 25 15:13:52 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.3@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:13:52 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.3@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:13:52.884389 21521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:52.876283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:43599 (local address 127.12.8.3:37527)
0925 15:13:52.876450 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:52.876471 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:52.876597 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:52.876876 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:52.876881 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:52.877260 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:13:52.877445 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:52.877457 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:52.878466 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:52.878472 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:52.879723 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:52.879736 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:52.880621 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:52.880628 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:52.880788 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:52.881319 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:13:52.881335 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:13:52.884120 (+ 2785us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.62:43599: Server oryx/127.12.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":54}
W20260925 15:13:52.884526 21258 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:13:52.876126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.3:37527 (local address 127.12.8.62:43599)
0925 15:13:52.876299 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:52.876305 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:52.876558 (+ 253us) server_negotiation.cc:407] Connection header received
0925 15:13:52.876699 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:52.876702 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:52.876754 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:52.876830 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:52.877590 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:52.878342 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:52.879916 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:52.880492 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:52.880621 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:52.884296 (+ 3675us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.3:37527: BlockingRecv error: recv got EOF from 127.12.8.3:37527 (error 108)
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":59}
Sep 25 15:13:52 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.3@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:13:52 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.3@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:13:52.892048 21521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:52.885009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:43599 (local address 127.12.8.3:60699)
0925 15:13:52.885154 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:52.885171 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:52.885263 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:52.885647 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:52.885650 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:52.885894 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:13:52.886029 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:52.886036 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:52.887050 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:52.887055 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:52.887911 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:52.887920 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:52.888579 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:52.888584 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:52.888739 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:52.889193 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:13:52.889209 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:13:52.891850 (+ 2641us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.62:43599: Server oryx/127.12.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":38}
W20260925 15:13:52.892334 21519 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 15:13:52.892532 21258 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:13:52.885161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.3:60699 (local address 127.12.8.62:43599)
0925 15:13:52.885391 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:52.885398 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:52.885417 (+ 19us) server_negotiation.cc:407] Connection header received
0925 15:13:52.885457 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:52.885462 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:52.885514 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:52.885627 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:52.886158 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:52.886912 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:52.888049 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:52.888443 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:52.888571 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:52.892023 (+ 3452us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.3:60699: BlockingRecv error: recv got EOF from 127.12.8.3:60699 (error 108)
Metrics: {"server-negotiator.queue_time_us":126}
Sep 25 15:13:53 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.1@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:13:53 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.1@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:13:53.024565 21524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:53.016114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:43599 (local address 127.12.8.1:39611)
0925 15:13:53.016800 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:53.016819 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:53.016941 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:53.017267 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:53.017271 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:53.017759 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:13:53.017974 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:53.017984 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:53.019052 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:53.019056 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:53.020091 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:53.020103 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:53.020806 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:53.020810 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:53.020937 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:53.021517 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:13:53.021535 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:13:53.024330 (+ 2795us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.62:43599: Server oryx/127.12.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":403,"threads_started":1}
W20260925 15:13:53.024712 21258 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:13:53.016171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.1:39611 (local address 127.12.8.62:43599)
0925 15:13:53.016384 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:53.016393 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:53.016908 (+ 515us) server_negotiation.cc:407] Connection header received
0925 15:13:53.017059 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:53.017063 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:53.017129 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:53.017219 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:53.018138 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:53.018913 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:53.020228 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:53.020648 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:53.020738 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:53.024512 (+ 3774us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.1:39611: BlockingRecv error: recv got EOF from 127.12.8.1:39611 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 25 15:13:53 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.2@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:13:53 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.2@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:13:53.462518 21525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:53.453911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:43599 (local address 127.12.8.2:49983)
0925 15:13:53.454396 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:53.454423 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:53.454571 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:53.454936 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:53.454939 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:53.455395 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:13:53.455623 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:53.455632 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:53.456671 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:53.456675 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:53.457900 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:53.457917 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:53.458817 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:53.458821 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:53.458940 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:53.459497 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:13:53.459514 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:13:53.462290 (+ 2776us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.62:43599: Server oryx/127.12.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":278,"threads_started":1}
W20260925 15:13:53.462672 21258 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:13:53.453937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.2:49983 (local address 127.12.8.62:43599)
0925 15:13:53.454158 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:53.454166 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:53.454509 (+ 343us) server_negotiation.cc:407] Connection header received
0925 15:13:53.454718 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:53.454722 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:53.454781 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:53.454900 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:53.455751 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:53.456536 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:53.458066 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:53.458666 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:53.458758 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:53.462436 (+ 3678us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.2:49983: BlockingRecv error: recv got EOF from 127.12.8.2:49983 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 25 15:13:53 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.3@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:13:53 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.3@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:13:53.907559 21526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:53.892972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:43599 (local address 127.12.8.3:36533)
0925 15:13:53.893474 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:53.893489 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:53.893651 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:53.893980 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:53.893985 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:53.894534 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:13:53.894740 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:53.894749 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:53.896109 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:53.896114 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:53.897313 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:53.897324 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:53.898035 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:53.898041 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:53.898268 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:53.904343 (+ 6075us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:13:53.904361 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:13:53.907272 (+ 2911us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.62:43599: Server oryx/127.12.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":269,"threads_started":1}
W20260925 15:13:53.907778 21258 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:13:53.893046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.3:36533 (local address 127.12.8.62:43599)
0925 15:13:53.893319 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:53.893326 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:53.893577 (+ 251us) server_negotiation.cc:407] Connection header received
0925 15:13:53.893778 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:53.893783 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:53.893850 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:53.893951 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:53.894884 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:53.895925 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:53.897458 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:53.897861 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:53.897998 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:53.907518 (+ 9520us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.3:36533: BlockingRecv error: recv got EOF from 127.12.8.3:36533 (error 108)
Metrics: {"server-negotiator.queue_time_us":113}
Sep 25 15:13:54 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.1@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:13:54 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.1@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:13:54.033624 21527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:54.025536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:43599 (local address 127.12.8.1:57811)
0925 15:13:54.026007 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:54.026023 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:54.026248 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:54.026598 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:54.026602 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:54.027013 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:13:54.027167 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:54.027176 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:54.028248 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:54.028254 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:54.029155 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:54.029166 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:54.029883 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:54.029889 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:54.030052 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:54.030647 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:13:54.030664 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:13:54.033395 (+ 2731us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.62:43599: Server oryx/127.12.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":248,"threads_started":1}
W20260925 15:13:54.033824 21258 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:13:54.025576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.1:57811 (local address 127.12.8.62:43599)
0925 15:13:54.025780 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:54.025788 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:54.026127 (+ 339us) server_negotiation.cc:407] Connection header received
0925 15:13:54.026388 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:54.026393 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:54.026464 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:54.026573 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:54.027306 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:54.028068 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:54.029302 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:54.029724 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:54.029867 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:54.033566 (+ 3699us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.1:57811: BlockingRecv error: recv got EOF from 127.12.8.1:57811 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 25 15:13:54 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.2@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:13:54 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.2@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:13:54.471848 21528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:54.463575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:43599 (local address 127.12.8.2:45953)
0925 15:13:54.464106 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:54.464140 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:54.464304 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:54.464627 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:54.464632 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:54.465048 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:13:54.465199 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:54.465208 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:54.466252 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:54.466258 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:54.467150 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:54.467160 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:54.467868 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:54.467874 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:54.468035 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:54.468729 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:13:54.468747 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:13:54.471483 (+ 2736us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.62:43599: Server oryx/127.12.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":266,"threads_started":1}
W20260925 15:13:54.471920 21258 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:13:54.463653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.2:45953 (local address 127.12.8.62:43599)
0925 15:13:54.463888 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:54.463895 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:54.464227 (+ 332us) server_negotiation.cc:407] Connection header received
0925 15:13:54.464418 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:54.464423 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:54.464494 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:54.464595 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:54.465342 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:54.466068 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:54.467292 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:54.467711 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:54.467837 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:54.471709 (+ 3872us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.2:45953: BlockingRecv error: recv got EOF from 127.12.8.2:45953 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 25 15:13:54 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.3@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:13:54 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.3@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:13:54.916594 21529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:54.908638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:43599 (local address 127.12.8.3:55125)
0925 15:13:54.909138 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:54.909154 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:54.909325 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:54.909731 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:54.909735 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:54.910228 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:13:54.910422 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:54.910438 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:54.911448 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:54.911452 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:54.912427 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:54.912439 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:54.913100 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:54.913106 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:54.913234 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:54.913779 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:13:54.913796 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:13:54.916367 (+ 2571us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.62:43599: Server oryx/127.12.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":274,"threads_started":1}
W20260925 15:13:54.916790 21258 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:13:54.908819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.3:55125 (local address 127.12.8.62:43599)
0925 15:13:54.909023 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:54.909031 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:54.909271 (+ 240us) server_negotiation.cc:407] Connection header received
0925 15:13:54.909455 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:54.909460 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:54.909573 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:54.909677 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:54.910599 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:54.911325 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:54.912542 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:54.912953 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:54.913047 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:54.916534 (+ 3487us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.3:55125: BlockingRecv error: recv got EOF from 127.12.8.3:55125 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 25 15:13:55 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.1@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:13:55 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.1@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:13:55.042932 21530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:55.034592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:43599 (local address 127.12.8.1:53199)
0925 15:13:55.035101 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:55.035119 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:55.035235 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:55.035518 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:55.035521 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:55.035863 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:13:55.036027 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:55.036036 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:55.037019 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:55.037022 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:55.037911 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:55.037922 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:55.039092 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:55.039099 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:55.039298 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:55.039974 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:13:55.039999 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:13:55.042689 (+ 2690us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.62:43599: Server oryx/127.12.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":285,"threads_started":1}
W20260925 15:13:55.043097 21258 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:13:55.034789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.1:53199 (local address 127.12.8.62:43599)
0925 15:13:55.034972 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:55.034976 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:55.035197 (+ 221us) server_negotiation.cc:407] Connection header received
0925 15:13:55.035331 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:55.035335 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:55.035393 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:55.035475 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:55.036173 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:55.036894 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:55.038148 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:55.038914 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:55.039057 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:55.042864 (+ 3807us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.1:53199: BlockingRecv error: recv got EOF from 127.12.8.1:53199 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 25 15:13:55 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.2@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:13:55 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.2@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:13:55.481576 21531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:55.472837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:43599 (local address 127.12.8.2:39225)
0925 15:13:55.473368 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:55.473384 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:55.473540 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:55.473905 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:55.473909 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:55.474317 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:13:55.474493 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:55.474502 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:55.475534 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:55.475540 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:55.476450 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:55.476461 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:55.477165 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:55.477170 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:55.477350 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:55.478111 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:13:55.478133 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:13:55.481221 (+ 3088us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.62:43599: Server oryx/127.12.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":308,"threads_started":1}
W20260925 15:13:55.481634 21258 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:13:55.472967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.2:39225 (local address 127.12.8.62:43599)
0925 15:13:55.473224 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:55.473230 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:55.473471 (+ 241us) server_negotiation.cc:407] Connection header received
0925 15:13:55.473695 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:55.473707 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:55.473767 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:55.473874 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:55.474647 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:55.475382 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:55.476594 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:55.477011 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:55.477152 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:55.481408 (+ 4256us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.2:39225: BlockingRecv error: recv got EOF from 127.12.8.2:39225 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 25 15:13:55 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.3@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:13:55 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.3@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:13:55.926338 21532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:55.917636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:43599 (local address 127.12.8.3:42433)
0925 15:13:55.918104 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:55.918121 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:55.918304 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:55.918775 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:55.918779 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:55.919133 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:13:55.919295 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:55.919303 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:55.920464 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:55.920468 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:55.921476 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:55.921490 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:55.922363 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:55.922369 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:55.922539 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:55.923234 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:13:55.923252 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:13:55.926034 (+ 2782us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.62:43599: Server oryx/127.12.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":256,"threads_started":1}
W20260925 15:13:55.926713 21258 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:13:55.917692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.3:42433 (local address 127.12.8.62:43599)
0925 15:13:55.917926 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:55.917933 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:55.918269 (+ 336us) server_negotiation.cc:407] Connection header received
0925 15:13:55.918504 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:55.918510 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:55.918603 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:55.918699 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:55.919496 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:55.920325 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:55.921625 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:55.922155 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:55.922342 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:55.926420 (+ 4078us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.3:42433: BlockingRecv error: recv got EOF from 127.12.8.3:42433 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 25 15:13:56 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.1@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:13:56 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.1@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:13:56.051851 21533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:56.043775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:43599 (local address 127.12.8.1:37973)
0925 15:13:56.044293 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:56.044309 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:56.044483 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:56.044812 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:56.044816 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:56.045195 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:13:56.045341 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:56.045349 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:56.046536 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:56.046542 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:56.047552 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:56.047563 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:56.048307 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:56.048311 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:56.048443 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:56.049021 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:13:56.049038 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:13:56.051636 (+ 2598us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.62:43599: Server oryx/127.12.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":287,"threads_started":1}
W20260925 15:13:56.052067 21258 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:13:56.043906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.1:37973 (local address 127.12.8.62:43599)
0925 15:13:56.044162 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:56.044169 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:56.044395 (+ 226us) server_negotiation.cc:407] Connection header received
0925 15:13:56.044608 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:56.044614 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:56.044684 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:56.044784 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:56.045527 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:56.046338 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:56.047702 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:56.048185 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:56.048284 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:56.051816 (+ 3532us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.1:37973: BlockingRecv error: recv got EOF from 127.12.8.1:37973 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
Sep 25 15:13:56 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.2@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:13:56 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.2@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:13:56.490587 21534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:56.482611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:43599 (local address 127.12.8.2:44899)
0925 15:13:56.483109 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:56.483125 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:56.483235 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:56.483536 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:56.483539 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:56.483898 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:13:56.484049 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:56.484057 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:56.485010 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:56.485013 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:56.485976 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:56.485987 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:56.486778 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:56.486784 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:56.486919 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:56.487478 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:13:56.487496 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:13:56.490232 (+ 2736us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.62:43599: Server oryx/127.12.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":269,"threads_started":1}
W20260925 15:13:56.490644 21258 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:13:56.482674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.2:44899 (local address 127.12.8.62:43599)
0925 15:13:56.482865 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:56.482870 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:56.483199 (+ 329us) server_negotiation.cc:407] Connection header received
0925 15:13:56.483330 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:56.483333 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:56.483405 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:56.483484 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:56.484181 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:56.484882 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:56.486104 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:56.486579 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:56.486728 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:56.490422 (+ 3694us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.2:44899: BlockingRecv error: recv got EOF from 127.12.8.2:44899 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 25 15:13:56 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.3@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:13:56 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.3@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:13:56.935587 21535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:56.927368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:43599 (local address 127.12.8.3:43053)
0925 15:13:56.927875 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:56.927890 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:56.928051 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:56.928415 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:56.928419 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:56.928808 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:13:56.928966 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:56.928974 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:56.929977 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:56.929985 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:56.931079 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:56.931092 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:56.931774 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:56.931779 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:56.931963 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:56.932528 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:13:56.932544 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:13:56.935367 (+ 2823us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.62:43599: Server oryx/127.12.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":295,"threads_started":1}
W20260925 15:13:56.935819 21258 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:13:56.927471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.3:43053 (local address 127.12.8.62:43599)
0925 15:13:56.927687 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:56.927696 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:56.927981 (+ 285us) server_negotiation.cc:407] Connection header received
0925 15:13:56.928188 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:56.928200 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:56.928271 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:56.928378 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:56.929113 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:56.929850 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:56.931226 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:56.931631 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:56.931765 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:56.935580 (+ 3815us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.3:43053: BlockingRecv error: recv got EOF from 127.12.8.3:43053 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 25 15:13:57 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.1@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:13:57 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.1@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:13:57.061357 21536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:57.052799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:43599 (local address 127.12.8.1:41275)
0925 15:13:57.053320 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:57.053341 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:57.053495 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:57.053815 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:57.053819 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:57.054287 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:13:57.054456 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:57.054464 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:57.055469 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:57.055474 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:57.056437 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:57.056448 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:57.057360 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:57.057366 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:57.057544 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:57.058153 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:13:57.058170 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:13:57.061010 (+ 2840us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.62:43599: Server oryx/127.12.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":262,"threads_started":1}
W20260925 15:13:57.061386 21258 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:13:57.052861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.1:41275 (local address 127.12.8.62:43599)
0925 15:13:57.053111 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:57.053120 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:57.053430 (+ 310us) server_negotiation.cc:407] Connection header received
0925 15:13:57.053627 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:57.053631 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:57.053688 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:57.053791 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:57.054624 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:57.055324 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:57.056602 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:57.057214 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:57.057339 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:57.061189 (+ 3850us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.1:41275: BlockingRecv error: recv got EOF from 127.12.8.1:41275 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 25 15:13:57 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.2@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:13:57 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.2@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:13:57.505538 21537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:57.491716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:43599 (local address 127.12.8.2:51517)
0925 15:13:57.492370 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:57.492386 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:57.492498 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:57.492841 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:57.492844 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:57.493203 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:13:57.493366 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:57.493374 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:57.494852 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:57.494856 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:57.495856 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:57.495868 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:57.496941 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:57.496948 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:57.497261 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:57.502657 (+ 5396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:13:57.502674 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:13:57.505300 (+ 2626us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.62:43599: Server oryx/127.12.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":381,"threads_started":1}
W20260925 15:13:57.505697 21258 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:13:57.491903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.2:51517 (local address 127.12.8.62:43599)
0925 15:13:57.492129 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:57.492135 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:57.492483 (+ 348us) server_negotiation.cc:407] Connection header received
0925 15:13:57.492624 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:57.492630 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:57.492709 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:57.492811 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:57.493495 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:57.494702 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:57.496015 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:57.496768 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:57.496884 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:57.505475 (+ 8591us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.2:51517: BlockingRecv error: recv got EOF from 127.12.8.2:51517 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 25 15:13:57 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.3@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:13:57 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.3@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:13:57.945358 21538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:57.936623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:43599 (local address 127.12.8.3:42179)
0925 15:13:57.937180 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:57.937196 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:57.937302 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:57.937681 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:57.937685 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:57.938171 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:13:57.938458 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:57.938471 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:57.939785 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:57.939788 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:57.940708 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:57.940718 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:57.941415 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:57.941420 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:57.941565 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:57.942136 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:13:57.942153 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:13:57.945000 (+ 2847us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.62:43599: Server oryx/127.12.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":291,"threads_started":1}
W20260925 15:13:57.945416 21258 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:13:57.936678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.3:42179 (local address 127.12.8.62:43599)
0925 15:13:57.936835 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:57.936840 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:57.937267 (+ 427us) server_negotiation.cc:407] Connection header received
0925 15:13:57.937443 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:57.937447 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:57.937520 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:57.937603 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:57.938629 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:57.939661 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:57.940842 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:57.941255 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:57.941394 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:57.945184 (+ 3790us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.3:42179: BlockingRecv error: recv got EOF from 127.12.8.3:42179 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 25 15:13:58 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.1@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:13:58 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.1@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:13:58.070394 21539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:58.062297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:43599 (local address 127.12.8.1:58625)
0925 15:13:58.062867 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:58.062883 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:58.063005 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:58.063281 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:58.063284 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:58.063652 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:13:58.063798 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:58.063806 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:58.064792 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:58.064796 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:58.065691 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:58.065702 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:58.066442 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:58.066446 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:58.066586 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:58.067118 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:13:58.067135 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:13:58.070125 (+ 2990us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.62:43599: Server oryx/127.12.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":340,"threads_started":1}
W20260925 15:13:58.070613 21258 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:13:58.062321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.1:58625 (local address 127.12.8.62:43599)
0925 15:13:58.062472 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:58.062477 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:58.062948 (+ 471us) server_negotiation.cc:407] Connection header received
0925 15:13:58.063097 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:58.063101 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:58.063160 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:58.063240 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:58.063915 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:58.064668 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:58.065809 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:58.066306 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:58.066397 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:58.070368 (+ 3971us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.1:58625: BlockingRecv error: recv got EOF from 127.12.8.1:58625 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 25 15:13:58 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.2@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:13:58 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.2@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:13:58.514739 21540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:58.506629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:43599 (local address 127.12.8.2:53137)
0925 15:13:58.507151 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:58.507168 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:58.507281 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:58.507566 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:58.507569 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:58.507934 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:13:58.508088 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:58.508096 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:58.509060 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:58.509064 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:58.509969 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:58.509981 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:58.510730 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:58.510735 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:58.510883 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:58.511472 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:13:58.511496 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:13:58.514391 (+ 2895us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.62:43599: Server oryx/127.12.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":291,"threads_started":1}
W20260925 15:13:58.514847 21258 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:13:58.506658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.2:53137 (local address 127.12.8.62:43599)
0925 15:13:58.506837 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:58.506842 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:58.507241 (+ 399us) server_negotiation.cc:407] Connection header received
0925 15:13:58.507377 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:58.507380 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:58.507438 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:58.507518 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:58.508219 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:58.508916 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:58.510095 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:58.510575 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:58.510666 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:58.514610 (+ 3944us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.2:53137: BlockingRecv error: recv got EOF from 127.12.8.2:53137 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 25 15:13:58 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.3@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:13:58 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.3@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:13:58.954991 21541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:58.946412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:43599 (local address 127.12.8.3:42329)
0925 15:13:58.946971 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:58.946988 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:58.947099 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:58.947421 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:58.947426 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:58.947838 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:13:58.948001 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:58.948011 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:58.949101 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:58.949107 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:58.949999 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:58.950009 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:58.950806 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:58.950812 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:58.950989 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:58.951587 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:13:58.951603 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:13:58.954721 (+ 3118us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.62:43599: Server oryx/127.12.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":322,"threads_started":1}
W20260925 15:13:58.955168 21258 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:13:58.946482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.3:42329 (local address 127.12.8.62:43599)
0925 15:13:58.946646 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:58.946651 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:58.947059 (+ 408us) server_negotiation.cc:407] Connection header received
0925 15:13:58.947188 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:58.947192 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:58.947263 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:58.947380 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:58.948147 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:58.948953 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:58.950172 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:58.950643 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:58.950790 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:58.954938 (+ 4148us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.3:42329: BlockingRecv error: recv got EOF from 127.12.8.3:42329 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 25 15:13:59 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.1@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:13:59 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.1@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:13:59.079890 21542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:59.071301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:43599 (local address 127.12.8.1:47609)
0925 15:13:59.071965 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:59.071982 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:59.072097 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:59.072383 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:59.072387 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:59.072761 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:13:59.072967 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:59.072976 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:59.074310 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:59.074315 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:59.075327 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:59.075341 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:59.076041 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:59.076045 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:59.076157 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:59.076716 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:13:59.076733 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:13:59.079649 (+ 2916us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.62:43599: Server oryx/127.12.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":292,"threads_started":1}
W20260925 15:13:59.080209 21258 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:13:59.071362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.1:47609 (local address 127.12.8.62:43599)
0925 15:13:59.071545 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:59.071550 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:59.072051 (+ 501us) server_negotiation.cc:407] Connection header received
0925 15:13:59.072191 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:59.072195 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:59.072252 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:59.072332 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:59.073118 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:59.074121 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:59.075483 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:59.075903 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:59.076001 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:59.079859 (+ 3858us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.1:47609: BlockingRecv error: recv got EOF from 127.12.8.1:47609 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 25 15:13:59 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.2@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:13:59 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.2@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:13:59.523587 21543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:59.515624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:43599 (local address 127.12.8.2:53273)
0925 15:13:59.516442 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:59.516457 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:59.516566 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:59.516864 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:59.516867 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:59.517244 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:13:59.517408 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:59.517416 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:59.518415 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:59.518418 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:59.519300 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:59.519311 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:59.519946 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:59.519950 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:59.520061 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:59.520682 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:13:59.520699 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:13:59.523363 (+ 2664us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.62:43599: Server oryx/127.12.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":671,"thread_start_us":507,"threads_started":1}
W20260925 15:13:59.523782 21258 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:13:59.515720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.2:53273 (local address 127.12.8.62:43599)
0925 15:13:59.515994 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:59.516003 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:59.516541 (+ 538us) server_negotiation.cc:407] Connection header received
0925 15:13:59.516665 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:59.516668 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:59.516740 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:59.516821 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:59.517563 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:59.518304 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:59.519417 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:59.519818 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:59.519907 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:59.523551 (+ 3644us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.2:53273: BlockingRecv error: recv got EOF from 127.12.8.2:53273 (error 108)
Metrics: {"server-negotiator.queue_time_us":112}
Sep 25 15:13:59 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.3@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:13:59 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.3@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:13:59.964160 21544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:13:59.956099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:43599 (local address 127.12.8.3:42091)
0925 15:13:59.956658 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0925 15:13:59.956674 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:13:59.956782 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:13:59.957102 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:13:59.957105 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:13:59.957583 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:13:59.957748 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:59.957758 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:59.958784 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:59.958788 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:59.959809 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:13:59.959825 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:13:59.960538 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:13:59.960542 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:13:59.960648 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:59.961265 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:13:59.961281 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:13:59.963926 (+ 2645us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.62:43599: Server oryx/127.12.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":238,"threads_started":1}
W20260925 15:13:59.964401 21258 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:13:59.956161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.3:42091 (local address 127.12.8.62:43599)
0925 15:13:59.956383 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0925 15:13:59.956393 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:13:59.956750 (+ 357us) server_negotiation.cc:407] Connection header received
0925 15:13:59.956891 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:13:59.956895 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:13:59.956984 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:13:59.957064 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:13:59.957864 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:59.958642 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:59.959952 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:13:59.960408 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:13:59.960502 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:13:59.964141 (+ 3639us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.3:42091: BlockingRecv error: recv got EOF from 127.12.8.3:42091 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 25 15:14:00 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.1@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:14:00 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.1@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:14:00.088937 21545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:14:00.080773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:43599 (local address 127.12.8.1:45363)
0925 15:14:00.081218 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0925 15:14:00.081234 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:14:00.081344 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:14:00.081631 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:14:00.081634 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:14:00.082045 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:14:00.082230 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:14:00.082239 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:14:00.083220 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:14:00.083224 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:14:00.084124 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:14:00.084134 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:14:00.084853 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:14:00.084858 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:14:00.084983 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:14:00.085651 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:14:00.085669 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:14:00.088611 (+ 2942us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.62:43599: Server oryx/127.12.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":232,"threads_started":1}
W20260925 15:14:00.089049 21258 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:14:00.080953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.1:45363 (local address 127.12.8.62:43599)
0925 15:14:00.081179 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0925 15:14:00.081190 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:14:00.081308 (+ 118us) server_negotiation.cc:407] Connection header received
0925 15:14:00.081450 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:14:00.081453 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:14:00.081512 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:14:00.081591 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:14:00.082356 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:14:00.083100 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:14:00.084249 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:14:00.084694 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:14:00.084788 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:14:00.088805 (+ 4017us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.1:45363: BlockingRecv error: recv got EOF from 127.12.8.1:45363 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Sep 25 15:14:00 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.2@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:14:00 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.2@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:14:00.532881 21546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:14:00.524568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:43599 (local address 127.12.8.2:42725)
0925 15:14:00.525089 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0925 15:14:00.525105 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:14:00.525217 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:14:00.525534 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:14:00.525537 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:14:00.525943 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:14:00.526095 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:14:00.526103 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:14:00.527323 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:14:00.527327 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:14:00.528209 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:14:00.528222 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:14:00.528936 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:14:00.528941 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:14:00.529068 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:14:00.529664 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:14:00.529680 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:14:00.532632 (+ 2952us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.62:43599: Server oryx/127.12.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":286,"threads_started":1}
W20260925 15:14:00.532970 21258 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:14:00.524615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.2:42725 (local address 127.12.8.62:43599)
0925 15:14:00.524791 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 15:14:00.524796 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:14:00.525180 (+ 384us) server_negotiation.cc:407] Connection header received
0925 15:14:00.525329 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:14:00.525332 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:14:00.525410 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:14:00.525482 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:14:00.526255 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:14:00.527144 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:14:00.528348 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:14:00.528771 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:14:00.528859 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:14:00.532782 (+ 3923us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.2:42725: BlockingRecv error: recv got EOF from 127.12.8.2:42725 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 25 15:14:00 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.3@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:14:00 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.3@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:14:00.973062 21547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:14:00.965047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:43599 (local address 127.12.8.3:49691)
0925 15:14:00.965543 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0925 15:14:00.965557 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 15:14:00.965713 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:14:00.966088 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:14:00.966093 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:14:00.966526 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:14:00.966671 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:14:00.966679 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:14:00.967671 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:14:00.967676 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:14:00.968607 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:14:00.968617 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:14:00.969406 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:14:00.969412 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:14:00.969592 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:14:00.970168 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:14:00.970228 (+ 60us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:14:00.972847 (+ 2619us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.62:43599: Server oryx/127.12.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":263,"threads_started":1}
W20260925 15:14:00.973270 21258 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:14:00.965114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.3:49691 (local address 127.12.8.62:43599)
0925 15:14:00.965418 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0925 15:14:00.965424 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:14:00.965639 (+ 215us) server_negotiation.cc:407] Connection header received
0925 15:14:00.965871 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:14:00.965883 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:14:00.965954 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:14:00.966056 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:14:00.966812 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:14:00.967522 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:14:00.968757 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:14:00.969216 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:14:00.969352 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:14:00.973042 (+ 3690us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.3:49691: BlockingRecv error: recv got EOF from 127.12.8.3:49691 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 25 15:14:01 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.1@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:14:01 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.1@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:14:01.098730 21548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:14:01.089965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:43599 (local address 127.12.8.1:46591)
0925 15:14:01.090544 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0925 15:14:01.090561 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 15:14:01.090713 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:14:01.091051 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:14:01.091055 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:14:01.091479 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:14:01.091641 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:14:01.091650 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:14:01.092651 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:14:01.092657 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:14:01.093586 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:14:01.093597 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:14:01.094416 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:14:01.094424 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:14:01.094611 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:14:01.095312 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:14:01.095344 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:14:01.098497 (+ 3153us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.62:43599: Server oryx/127.12.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":374,"threads_started":1}
W20260925 15:14:01.098964 21258 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:14:01.090046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.1:46591 (local address 127.12.8.62:43599)
0925 15:14:01.090303 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0925 15:14:01.090310 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:14:01.090646 (+ 336us) server_negotiation.cc:407] Connection header received
0925 15:14:01.090854 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:14:01.090858 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:14:01.090916 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:14:01.091021 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:14:01.091793 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:14:01.092510 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:14:01.093725 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:14:01.094165 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:14:01.094347 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:14:01.098696 (+ 4349us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.1:46591: BlockingRecv error: recv got EOF from 127.12.8.1:46591 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Sep 25 15:14:01 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.2@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:14:01 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.2@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:14:01.541878 21549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:14:01.533724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:43599 (local address 127.12.8.2:53295)
0925 15:14:01.534217 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0925 15:14:01.534244 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 15:14:01.534387 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:14:01.534678 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:14:01.534681 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:14:01.535065 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:14:01.535223 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:14:01.535232 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:14:01.536196 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:14:01.536199 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:14:01.537132 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:14:01.537143 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:14:01.537771 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:14:01.537775 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:14:01.537888 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:14:01.538461 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:14:01.538480 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:14:01.541511 (+ 3031us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.62:43599: Server oryx/127.12.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":257,"threads_started":1}
W20260925 15:14:01.541978 21258 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:14:01.533867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.2:53295 (local address 127.12.8.62:43599)
0925 15:14:01.534081 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0925 15:14:01.534088 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:14:01.534353 (+ 265us) server_negotiation.cc:407] Connection header received
0925 15:14:01.534486 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:14:01.534489 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:14:01.534548 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:14:01.534625 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:14:01.535369 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:14:01.536082 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:14:01.537250 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:14:01.537652 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:14:01.537741 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:14:01.541748 (+ 4007us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.2:53295: BlockingRecv error: recv got EOF from 127.12.8.2:53295 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 25 15:14:01 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.3@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:14:01 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.3@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:14:01.983585 21550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:14:01.974136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:43599 (local address 127.12.8.3:33233)
0925 15:14:01.974813 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0925 15:14:01.974836 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 15:14:01.974986 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:14:01.975495 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:14:01.975505 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:14:01.976052 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:14:01.976273 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:14:01.976283 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:14:01.977772 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:14:01.977776 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:14:01.978709 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:14:01.978720 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:14:01.979427 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:14:01.979431 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:14:01.979550 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:14:01.980106 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:14:01.980122 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:14:01.983211 (+ 3089us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.62:43599: Server oryx/127.12.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":328,"threads_started":1}
W20260925 15:14:01.983700 21258 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:14:01.974274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.3:33233 (local address 127.12.8.62:43599)
0925 15:14:01.974534 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0925 15:14:01.974543 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:14:01.974931 (+ 388us) server_negotiation.cc:407] Connection header received
0925 15:14:01.975109 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:14:01.975114 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:14:01.975218 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:14:01.975326 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:14:01.976419 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:14:01.977636 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:14:01.978846 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:14:01.979285 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:14:01.979403 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:14:01.983457 (+ 4054us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.3:33233: BlockingRecv error: recv got EOF from 127.12.8.3:33233 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Sep 25 15:14:02 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.1@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:14:02 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.1@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:14:02.108282 21551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:14:02.099640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:43599 (local address 127.12.8.1:39457)
0925 15:14:02.100166 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0925 15:14:02.100181 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:14:02.100334 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:14:02.100671 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:14:02.100676 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:14:02.101060 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:14:02.101197 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:14:02.101206 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:14:02.102226 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:14:02.102233 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:14:02.103158 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:14:02.103168 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:14:02.103885 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:14:02.103891 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:14:02.104080 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:14:02.104643 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:14:02.104660 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:14:02.107960 (+ 3300us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.62:43599: Server oryx/127.12.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":310,"threads_started":1}
W20260925 15:14:02.108484 21258 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:14:02.099756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.1:39457 (local address 127.12.8.62:43599)
0925 15:14:02.100014 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0925 15:14:02.100021 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:14:02.100263 (+ 242us) server_negotiation.cc:407] Connection header received
0925 15:14:02.100465 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:14:02.100471 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:14:02.100542 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:14:02.100643 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:14:02.101344 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:14:02.102064 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:14:02.103302 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:14:02.103736 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:14:02.103873 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:14:02.108158 (+ 4285us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.1:39457: BlockingRecv error: recv got EOF from 127.12.8.1:39457 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 25 15:14:02 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.2@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:14:02 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.2@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:14:02.555521 21552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:14:02.542963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:43599 (local address 127.12.8.2:46005)
0925 15:14:02.543475 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0925 15:14:02.543493 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 15:14:02.543610 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:14:02.543903 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:14:02.543907 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:14:02.544285 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:14:02.544452 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:14:02.544461 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:14:02.545461 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:14:02.545465 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:14:02.546541 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:14:02.546555 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:14:02.547204 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:14:02.547208 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:14:02.547328 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:14:02.552131 (+ 4803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:14:02.552149 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:14:02.555286 (+ 3137us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.62:43599: Server oryx/127.12.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":289,"threads_started":1}
W20260925 15:14:02.555812 21258 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:14:02.543127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.2:46005 (local address 127.12.8.62:43599)
0925 15:14:02.543355 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0925 15:14:02.543362 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:14:02.543576 (+ 214us) server_negotiation.cc:407] Connection header received
0925 15:14:02.543710 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:14:02.543713 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:14:02.543771 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:14:02.543850 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:14:02.544584 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:14:02.545311 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:14:02.546672 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:14:02.547075 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:14:02.547170 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:14:02.555498 (+ 8328us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.2:46005: BlockingRecv error: recv got EOF from 127.12.8.2:46005 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Sep 25 15:14:02 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.3@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:14:02 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.3@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:14:02.993235 21553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:14:02.984625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:43599 (local address 127.12.8.3:34663)
0925 15:14:02.985288 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0925 15:14:02.985304 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:14:02.985472 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:14:02.985811 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:14:02.985816 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:14:02.986412 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:14:02.986561 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:14:02.986569 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:14:02.987582 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:14:02.987588 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:14:02.988509 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:14:02.988520 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:14:02.989241 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:14:02.989247 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:14:02.989413 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:14:02.990038 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:14:02.990056 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:14:02.992916 (+ 2860us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.62:43599: Server oryx/127.12.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":406,"threads_started":1}
W20260925 15:14:02.993391 21258 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:14:02.984829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.3:34663 (local address 127.12.8.62:43599)
0925 15:14:02.985113 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0925 15:14:02.985120 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:14:02.985389 (+ 269us) server_negotiation.cc:407] Connection header received
0925 15:14:02.985591 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:14:02.985596 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:14:02.985664 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:14:02.985779 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:14:02.986714 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:14:02.987439 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:14:02.988663 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:14:02.989077 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:14:02.989202 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:14:02.993143 (+ 3941us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.3:34663: BlockingRecv error: recv got EOF from 127.12.8.3:34663 (error 108)
Metrics: {"server-negotiator.queue_time_us":133}
Sep 25 15:14:03 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.1@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:14:03 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.1@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:14:03.118360 21554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:14:03.109417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:43599 (local address 127.12.8.1:44513)
0925 15:14:03.109963 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0925 15:14:03.109980 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 15:14:03.110092 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:14:03.110387 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:14:03.110390 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:14:03.110775 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:14:03.110964 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:14:03.110972 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:14:03.111891 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:14:03.111895 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:14:03.113000 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:14:03.113016 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:14:03.114305 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:14:03.114311 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:14:03.114467 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:14:03.115081 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:14:03.115098 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:14:03.117984 (+ 2886us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.62:43599: Server oryx/127.12.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":285,"threads_started":1}
W20260925 15:14:03.118422 21258 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:14:03.109507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.1:44513 (local address 127.12.8.62:43599)
0925 15:14:03.109679 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 15:14:03.109684 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:14:03.110049 (+ 365us) server_negotiation.cc:407] Connection header received
0925 15:14:03.110203 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:14:03.110206 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:14:03.110264 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:14:03.110344 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:14:03.111086 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:14:03.111781 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:14:03.113228 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:14:03.114071 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:14:03.114235 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:14:03.118176 (+ 3941us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.1:44513: BlockingRecv error: recv got EOF from 127.12.8.1:44513 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 25 15:14:03 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.2@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:14:03 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.2@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:14:03.564872 21555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:14:03.556588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:43599 (local address 127.12.8.2:43473)
0925 15:14:03.557175 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0925 15:14:03.557196 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 15:14:03.557332 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:14:03.557685 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:14:03.557690 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:14:03.558238 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:14:03.558440 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:14:03.558450 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:14:03.559562 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:14:03.559566 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:14:03.560518 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:14:03.560529 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:14:03.561168 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:14:03.561173 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:14:03.561284 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:14:03.561856 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:14:03.561879 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:14:03.564618 (+ 2739us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.62:43599: Server oryx/127.12.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":312,"threads_started":1}
W20260925 15:14:03.565029 21258 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:14:03.556649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.2:43473 (local address 127.12.8.62:43599)
0925 15:14:03.556865 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 15:14:03.556872 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:14:03.557331 (+ 459us) server_negotiation.cc:407] Connection header received
0925 15:14:03.557430 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:14:03.557436 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:14:03.557518 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:14:03.557598 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:14:03.558568 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:14:03.559438 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:14:03.560643 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:14:03.561033 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:14:03.561124 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:14:03.564801 (+ 3677us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.2:43473: BlockingRecv error: recv got EOF from 127.12.8.2:43473 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 25 15:14:03 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.3@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:14:04 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.3@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:14:04.003176 21556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:14:03.994426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:43599 (local address 127.12.8.3:45789)
0925 15:14:03.994939 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0925 15:14:03.994955 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:14:03.995065 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:14:03.995439 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:14:03.995443 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:14:03.995871 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:14:03.996098 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:14:03.996107 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:14:03.997257 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:14:03.997260 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:14:03.998144 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:14:03.998154 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:14:03.998936 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:14:03.998944 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:14:03.999109 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:14:03.999877 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:14:03.999901 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:14:04.002945 (+ 3044us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.62:43599: Server oryx/127.12.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":281,"threads_started":1}
W20260925 15:14:04.003384 21258 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:14:03.994475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.3:45789 (local address 127.12.8.62:43599)
0925 15:14:03.994647 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 15:14:03.994651 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:14:03.995033 (+ 382us) server_negotiation.cc:407] Connection header received
0925 15:14:03.995214 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:14:03.995218 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:14:03.995306 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:14:03.995395 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:14:03.996342 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:14:03.997144 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:14:03.998386 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:14:03.998787 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:14:03.998902 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:14:04.003153 (+ 4251us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.3:45789: BlockingRecv error: recv got EOF from 127.12.8.3:45789 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 25 15:14:04 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.1@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:14:04 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.1@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:14:04.127266 21557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:14:04.119406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:43599 (local address 127.12.8.1:35765)
0925 15:14:04.119949 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0925 15:14:04.119969 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 15:14:04.120110 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:14:04.120529 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:14:04.120533 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:14:04.120889 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:14:04.121052 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:14:04.121061 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:14:04.122018 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:14:04.122022 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:14:04.122928 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:14:04.122942 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:14:04.123620 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:14:04.123623 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:14:04.123744 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:14:04.124378 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:14:04.124395 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:14:04.127043 (+ 2648us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.62:43599: Server oryx/127.12.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":274,"threads_started":1}
W20260925 15:14:04.127470 21258 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:14:04.119595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.1:35765 (local address 127.12.8.62:43599)
0925 15:14:04.119816 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0925 15:14:04.119822 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:14:04.120056 (+ 234us) server_negotiation.cc:407] Connection header received
0925 15:14:04.120233 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:14:04.120239 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:14:04.120339 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:14:04.120454 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:14:04.121177 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:14:04.121899 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:14:04.123050 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:14:04.123501 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:14:04.123591 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:14:04.127218 (+ 3627us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.1:35765: BlockingRecv error: recv got EOF from 127.12.8.1:35765 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 25 15:14:04 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.2@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:14:04 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.2@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:14:04.574077 21558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:14:04.565750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:43599 (local address 127.12.8.2:51429)
0925 15:14:04.566304 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0925 15:14:04.566320 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:14:04.566443 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:14:04.566718 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:14:04.566721 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:14:04.567086 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:14:04.567248 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:14:04.567256 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:14:04.568199 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:14:04.568203 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:14:04.569222 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:14:04.569234 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:14:04.569951 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:14:04.569955 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:14:04.570085 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:14:04.570786 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:14:04.570826 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:14:04.573852 (+ 3026us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.62:43599: Server oryx/127.12.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":298,"threads_started":1}
W20260925 15:14:04.574250 21258 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:14:04.565768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.2:51429 (local address 127.12.8.62:43599)
0925 15:14:04.565944 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 15:14:04.565948 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:14:04.566388 (+ 440us) server_negotiation.cc:407] Connection header received
0925 15:14:04.566538 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:14:04.566541 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:14:04.566600 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:14:04.566679 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:14:04.567375 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:14:04.568070 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:14:04.569372 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:14:04.569822 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:14:04.569921 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:14:04.574031 (+ 4110us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.2:51429: BlockingRecv error: recv got EOF from 127.12.8.2:51429 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 25 15:14:05 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.3@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:14:05 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.3@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:14:05.012827 21559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:14:05.004169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:43599 (local address 127.12.8.3:38365)
0925 15:14:05.004750 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0925 15:14:05.004772 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 15:14:05.004882 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:14:05.005200 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:14:05.005204 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:14:05.005591 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:14:05.005749 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:14:05.005757 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:14:05.006861 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:14:05.006865 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:14:05.007762 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:14:05.007773 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:14:05.008616 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:14:05.008624 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:14:05.008769 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:14:05.009543 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:14:05.009568 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:14:05.012602 (+ 3034us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.62:43599: Server oryx/127.12.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":342,"threads_started":1}
W20260925 15:14:05.013182 21258 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:14:05.004223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.3:38365 (local address 127.12.8.62:43599)
0925 15:14:05.004520 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0925 15:14:05.004525 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:14:05.004851 (+ 326us) server_negotiation.cc:407] Connection header received
0925 15:14:05.004979 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:14:05.004983 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:14:05.005045 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:14:05.005128 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:14:05.005938 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:14:05.006716 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:14:05.007913 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:14:05.008475 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:14:05.008573 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:14:05.012886 (+ 4313us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.3:38365: BlockingRecv error: recv got EOF from 127.12.8.3:38365 (error 108)
Metrics: {"server-negotiator.queue_time_us":183}
Sep 25 15:14:05 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.1@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:14:05 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.1@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:14:05.136093 21560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:14:05.128137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:43599 (local address 127.12.8.1:59981)
0925 15:14:05.128705 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0925 15:14:05.128722 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 15:14:05.128835 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:14:05.129120 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:14:05.129123 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:14:05.129516 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:14:05.129688 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:14:05.129696 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:14:05.130700 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:14:05.130704 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:14:05.131607 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:14:05.131617 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:14:05.132286 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:14:05.132290 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:14:05.132417 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:14:05.133010 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:14:05.133027 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:14:05.135865 (+ 2838us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.62:43599: Server oryx/127.12.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":299,"threads_started":1}
W20260925 15:14:05.136323 21258 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:14:05.128197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.1:59981 (local address 127.12.8.62:43599)
0925 15:14:05.128395 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 15:14:05.128399 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:14:05.128798 (+ 399us) server_negotiation.cc:407] Connection header received
0925 15:14:05.128932 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:14:05.128936 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:14:05.128994 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:14:05.129071 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:14:05.129811 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:14:05.130582 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:14:05.131729 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:14:05.132140 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:14:05.132230 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:14:05.136055 (+ 3825us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.1:59981: BlockingRecv error: recv got EOF from 127.12.8.1:59981 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 25 15:14:05 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.2@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:14:05 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.2@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:14:05.583508 21561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:14:05.575150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:43599 (local address 127.12.8.2:57929)
0925 15:14:05.575731 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0925 15:14:05.575751 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 15:14:05.575888 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:14:05.576254 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:14:05.576259 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:14:05.576773 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:14:05.576949 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:14:05.576959 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:14:05.577947 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:14:05.577950 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:14:05.578898 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:14:05.578909 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:14:05.579629 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:14:05.579633 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:14:05.579757 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:14:05.580407 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:14:05.580424 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:14:05.583166 (+ 2742us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.62:43599: Server oryx/127.12.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":343,"threads_started":1}
W20260925 15:14:05.583571 21258 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:14:05.575299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.2:57929 (local address 127.12.8.62:43599)
0925 15:14:05.575485 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 15:14:05.575494 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:14:05.575866 (+ 372us) server_negotiation.cc:407] Connection header received
0925 15:14:05.576026 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:14:05.576033 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:14:05.576111 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:14:05.576212 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:14:05.577077 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:14:05.577807 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:14:05.579020 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:14:05.579465 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:14:05.579560 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:14:05.583351 (+ 3791us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.2:57929: BlockingRecv error: recv got EOF from 127.12.8.2:57929 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 25 15:14:06 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.3@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:14:06 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.3@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:14:06.022329 21562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:14:06.013927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:43599 (local address 127.12.8.3:42875)
0925 15:14:06.014498 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0925 15:14:06.014513 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:14:06.014673 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:14:06.015116 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:14:06.015124 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:14:06.015620 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:14:06.015851 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:14:06.015875 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:14:06.016876 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:14:06.016881 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:14:06.017833 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:14:06.017846 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:14:06.018578 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:14:06.018583 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:14:06.018726 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:14:06.019289 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:14:06.019307 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:14:06.022038 (+ 2731us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.62:43599: Server oryx/127.12.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":322,"threads_started":1}
W20260925 15:14:06.022505 21258 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:14:06.014017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.3:42875 (local address 127.12.8.62:43599)
0925 15:14:06.014254 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0925 15:14:06.014262 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:14:06.014605 (+ 343us) server_negotiation.cc:407] Connection header received
0925 15:14:06.014864 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:14:06.014869 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:14:06.014948 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:14:06.015058 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:14:06.016018 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:14:06.016721 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:14:06.017974 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:14:06.018417 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:14:06.018512 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:14:06.022305 (+ 3793us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.3:42875: BlockingRecv error: recv got EOF from 127.12.8.3:42875 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 25 15:14:06 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.1@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:14:06 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.1@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:14:06.145011 21563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:14:06.137052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:43599 (local address 127.12.8.1:43151)
0925 15:14:06.137557 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0925 15:14:06.137574 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 15:14:06.137686 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:14:06.138014 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:14:06.138017 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:14:06.138396 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:14:06.138546 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:14:06.138553 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:14:06.139527 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:14:06.139531 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:14:06.140452 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:14:06.140463 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:14:06.141110 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:14:06.141114 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:14:06.141232 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:14:06.141811 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:14:06.141827 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:14:06.144704 (+ 2877us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.62:43599: Server oryx/127.12.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":293,"threads_started":1}
W20260925 15:14:06.145282 21258 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:14:06.137157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.1:43151 (local address 127.12.8.62:43599)
0925 15:14:06.137438 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0925 15:14:06.137442 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:14:06.137648 (+ 206us) server_negotiation.cc:407] Connection header received
0925 15:14:06.137789 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:14:06.137793 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:14:06.137862 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:14:06.137970 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:14:06.138670 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:14:06.139402 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:14:06.140586 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:14:06.140988 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:14:06.141080 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:14:06.144991 (+ 3911us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.1:43151: BlockingRecv error: recv got EOF from 127.12.8.1:43151 (error 108)
Metrics: {"server-negotiator.queue_time_us":112}
Sep 25 15:14:06 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.2@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:14:06 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.2@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:14:06.592759 21564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:14:06.584458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:43599 (local address 127.12.8.2:57623)
0925 15:14:06.584939 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0925 15:14:06.584954 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:14:06.585110 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:14:06.585446 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:14:06.585451 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:14:06.585838 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:14:06.585999 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:14:06.586008 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:14:06.587086 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:14:06.587092 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:14:06.588012 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:14:06.588023 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:14:06.588749 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:14:06.588755 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:14:06.588929 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:14:06.589486 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:14:06.589502 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:14:06.592537 (+ 3035us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.62:43599: Server oryx/127.12.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":247,"threads_started":1}
W20260925 15:14:06.592954 21258 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:14:06.584547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.2:57623 (local address 127.12.8.62:43599)
0925 15:14:06.584770 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0925 15:14:06.584776 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:14:06.585036 (+ 260us) server_negotiation.cc:407] Connection header received
0925 15:14:06.585230 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:14:06.585236 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:14:06.585307 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:14:06.585423 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:14:06.586140 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:14:06.586942 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:14:06.588168 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:14:06.588602 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:14:06.588737 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:14:06.592719 (+ 3982us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.2:57623: BlockingRecv error: recv got EOF from 127.12.8.2:57623 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 25 15:14:07 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.3@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:14:07 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.3@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:14:07.031514 21565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:14:07.023330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:43599 (local address 127.12.8.3:34379)
0925 15:14:07.023839 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0925 15:14:07.023855 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:14:07.024013 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:14:07.024387 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:14:07.024391 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:14:07.024746 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:14:07.024902 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:14:07.024910 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:14:07.026004 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:14:07.026008 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:14:07.026968 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:14:07.026980 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:14:07.027679 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:14:07.027683 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:14:07.027810 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:14:07.028395 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:14:07.028412 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:14:07.031272 (+ 2860us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.62:43599: Server oryx/127.12.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":283,"threads_started":1}
W20260925 15:14:07.031774 21258 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:14:07.023410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.3:34379 (local address 127.12.8.62:43599)
0925 15:14:07.023681 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0925 15:14:07.023688 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:14:07.023966 (+ 278us) server_negotiation.cc:407] Connection header received
0925 15:14:07.024151 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:14:07.024155 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:14:07.024216 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:14:07.024300 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:14:07.025059 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:14:07.025853 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:14:07.027113 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:14:07.027542 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:14:07.027640 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:14:07.031505 (+ 3865us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.3:34379: BlockingRecv error: recv got EOF from 127.12.8.3:34379 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 25 15:14:07 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.1@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:14:07 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.1@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:14:07.154531 21566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:14:07.146114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:43599 (local address 127.12.8.1:48195)
0925 15:14:07.146632 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0925 15:14:07.146650 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 15:14:07.146762 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:14:07.147028 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:14:07.147031 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:14:07.147505 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:14:07.147671 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:14:07.147679 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:14:07.148717 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:14:07.148721 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:14:07.149687 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:14:07.149699 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:14:07.150420 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:14:07.150424 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:14:07.150537 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:14:07.151078 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:14:07.151100 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:14:07.154132 (+ 3032us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.62:43599: Server oryx/127.12.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":294,"threads_started":1}
W20260925 15:14:07.154604 21258 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:14:07.146207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.1:48195 (local address 127.12.8.62:43599)
0925 15:14:07.146479 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0925 15:14:07.146485 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:14:07.146734 (+ 249us) server_negotiation.cc:407] Connection header received
0925 15:14:07.146851 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:14:07.146854 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:14:07.146910 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:14:07.146989 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:14:07.147830 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:14:07.148587 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:14:07.149809 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:14:07.150298 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:14:07.150392 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:14:07.154349 (+ 3957us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.1:48195: BlockingRecv error: recv got EOF from 127.12.8.1:48195 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Sep 25 15:14:07 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.2@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:14:07 dist-test-slave-p468 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.8.2@KRBTEST.COM for oryx/127.12.8.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:14:07.606455 21567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:14:07.594006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:43599 (local address 127.12.8.2:47165)
0925 15:14:07.594678 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0925 15:14:07.594695 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 15:14:07.594815 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:14:07.595187 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:14:07.595190 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:14:07.595732 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:14:07.595899 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:14:07.595909 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:14:07.597021 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:14:07.597025 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:14:07.598045 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:14:07.598059 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:14:07.598801 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:14:07.598805 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:14:07.598915 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:14:07.602908 (+ 3993us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:14:07.602927 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:14:07.606020 (+ 3093us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.8.62:43599: Server oryx/127.12.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":435,"threads_started":1}
W20260925 15:14:07.606520 21258 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:14:07.594079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.2:47165 (local address 127.12.8.62:43599)
0925 15:14:07.594335 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0925 15:14:07.594344 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:14:07.594770 (+ 426us) server_negotiation.cc:407] Connection header received
0925 15:14:07.594937 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:14:07.594945 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:14:07.595024 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:14:07.595174 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:14:07.596023 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:14:07.596870 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:14:07.598265 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:14:07.598682 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:14:07.598774 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:14:07.606263 (+ 7489us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.8.2:47165: BlockingRecv error: recv got EOF from 127.12.8.2:47165 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 15:14:07.866657 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 21109
I20260925 15:14:07.879137 21569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:14:07.867720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53202 (local address 127.12.8.1:42053)
0925 15:14:07.868292 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0925 15:14:07.868302 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:14:07.868412 (+ 110us) server_negotiation.cc:407] Connection header received
0925 15:14:07.868594 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:14:07.868618 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:14:07.868919 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:14:07.869210 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:14:07.871146 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:14:07.872170 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:14:07.873297 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:14:07.873828 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:14:07.873933 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:14:07.876283 (+ 2350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:14:07.876292 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:14:07.876306 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:14:07.876350 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:14:07.878657 (+ 2307us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.12.8.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0925 15:14:07.878911 (+ 254us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53202: Request ticket server kudu/127.12.8.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":333,"threads_started":1}
W20260925 15:14:07.879276 12320 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 21109: Not authorized: Client connection negotiation failed: client connection to 127.12.8.1:42053: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.12.8.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260925 15:14:07.879518 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 21109
I20260925 15:14:07.895922 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 21260
I20260925 15:14:07.907709 21570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:14:07.897125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60284 (local address 127.12.8.2:41741)
0925 15:14:07.897725 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0925 15:14:07.897732 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:14:07.897762 (+ 30us) server_negotiation.cc:407] Connection header received
0925 15:14:07.897820 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:14:07.897827 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:14:07.898034 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:14:07.898236 (+ 202us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:14:07.899437 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:14:07.900477 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:14:07.901616 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:14:07.902137 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:14:07.902372 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:14:07.904740 (+ 2368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:14:07.904748 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:14:07.904760 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:14:07.904791 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:14:07.907181 (+ 2390us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.12.8.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0925 15:14:07.907426 (+ 245us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60284: Request ticket server kudu/127.12.8.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":309,"threads_started":1}
W20260925 15:14:07.907809 21570 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60284: Request ticket server kudu/127.12.8.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260925 15:14:07.907856 12320 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 21260: Not authorized: Client connection negotiation failed: client connection to 127.12.8.2:41741: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.12.8.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260925 15:14:07.908159 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 21260
I20260925 15:14:07.926136 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 21391
I20260925 15:14:07.937806 21571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:14:07.927150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45638 (local address 127.12.8.3:34059)
0925 15:14:07.927710 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0925 15:14:07.927717 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:14:07.927743 (+ 26us) server_negotiation.cc:407] Connection header received
0925 15:14:07.927808 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:14:07.927814 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:14:07.928027 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:14:07.928180 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:14:07.929396 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:14:07.930498 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:14:07.931613 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:14:07.932171 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:14:07.932298 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:14:07.934871 (+ 2573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:14:07.934880 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:14:07.934893 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:14:07.934930 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:14:07.937333 (+ 2403us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.12.8.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0925 15:14:07.937570 (+ 237us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45638: Request ticket server kudu/127.12.8.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":301,"threads_started":1}
W20260925 15:14:07.937925 21571 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45638: Request ticket server kudu/127.12.8.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260925 15:14:07.938004 12320 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 21391: Not authorized: Client connection negotiation failed: client connection to 127.12.8.3:34059: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.12.8.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260925 15:14:07.938256 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 21391
I20260925 15:14:07.954269 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 21040
I20260925 15:14:07.955569 21102 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:14:08.069036 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 21040
2026-09-25T15:14:08Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (20960 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-25T15:14:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:14:08Z Disabled control of system clock
I20260925 15:14:08.143513 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349162370577-12320-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349162370577-12320-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349162370577-12320-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.62:45349
--webserver_interface=127.12.8.62
--webserver_port=0
--builtin_ntp_servers=127.12.8.20:33925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.62:45349
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 15:14:08.460386 21576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:14:08.460659 21576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:14:08.460740 21576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:14:08.467897 21576 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:14:08.468022 21576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:14:08.468063 21576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:14:08.468113 21576 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:14:08.468158 21576 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:14:08.477885 21576 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:33925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349162370577-12320-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349162370577-12320-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.8.62:45349
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.62:45349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.8.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349162370577-12320-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:14:08.479218 21576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:14:08.481529 21576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:14:08.493134 21581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:14:08.493093 21582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:14:08.493165 21584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:14:08.493891 21576 server_base.cc:1034] running on GCE node
I20260925 15:14:08.494668 21576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:14:08.495966 21576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:14:08.497190 21576 hybrid_clock.cc:648] HybridClock initialized: now 1790349248497157 us; error 46 us; skew 500 ppm
I20260925 15:14:08.497524 21576 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:14:08.499732 21576 webserver.cc:460] Webserver started at http://127.12.8.62:46227/ using document root <none> and password file <none>
I20260925 15:14:08.500455 21576 fs_manager.cc:359] Metadata directory not provided
I20260925 15:14:08.500577 21576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:14:08.500893 21576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:14:08.503670 21576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349162370577-12320-0/minicluster-data/master-0/data/instance:
uuid: "e1a68f3941314c55a1d8fbe688770692"
format_stamp: "Formatted at 2026-09-25 15:14:08 on dist-test-slave-p468"
server_key: "f918d797c9c663caa93c1dfb2d9fa0e8"
server_key_iv: "dbea79728db3feb91c84ab63eabc64ed"
server_key_version: "encryptionkey@0"
I20260925 15:14:08.504433 21576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349162370577-12320-0/minicluster-data/master-0/wal/instance:
uuid: "e1a68f3941314c55a1d8fbe688770692"
format_stamp: "Formatted at 2026-09-25 15:14:08 on dist-test-slave-p468"
server_key: "f918d797c9c663caa93c1dfb2d9fa0e8"
server_key_iv: "dbea79728db3feb91c84ab63eabc64ed"
server_key_version: "encryptionkey@0"
I20260925 15:14:08.509300 21576 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260925 15:14:08.512735 21590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:14:08.514034 21576 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 15:14:08.514312 21576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349162370577-12320-0/minicluster-data/master-0/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349162370577-12320-0/minicluster-data/master-0/wal
uuid: "e1a68f3941314c55a1d8fbe688770692"
format_stamp: "Formatted at 2026-09-25 15:14:08 on dist-test-slave-p468"
server_key: "f918d797c9c663caa93c1dfb2d9fa0e8"
server_key_iv: "dbea79728db3feb91c84ab63eabc64ed"
server_key_version: "encryptionkey@0"
I20260925 15:14:08.514524 21576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349162370577-12320-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349162370577-12320-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349162370577-12320-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:14:08.536909 21576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:14:08.537788 21576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:14:08.538110 21576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:14:08.557272 21576 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.62:45349
I20260925 15:14:08.557292 21641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.62:45349 every 8 connection(s)
I20260925 15:14:08.558641 21576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
I20260925 15:14:08.561856 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 21576
I20260925 15:14:08.562034 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349162370577-12320-0/minicluster-data/master-0/wal/instance
I20260925 15:14:08.562597 21642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:14:08.562883 12320 external_mini_cluster.cc:1428] Setting key d332fdbde3ec49e0831637d107b58ac2
I20260925 15:14:08.576116 21642 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1a68f3941314c55a1d8fbe688770692: Bootstrap starting.
I20260925 15:14:08.576520 21645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:14:08.564870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55168 (local address 127.12.8.62:45349)
0925 15:14:08.566058 (+ 1188us) server_negotiation.cc:206] Beginning negotiation
0925 15:14:08.566069 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:14:08.566111 (+ 42us) server_negotiation.cc:407] Connection header received
0925 15:14:08.567624 (+ 1513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:14:08.567643 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:14:08.568092 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:14:08.568414 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:14:08.569060 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:14:08.570371 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:14:08.572193 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:14:08.572860 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:14:08.572996 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:14:08.574263 (+ 1267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:14:08.574272 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:14:08.574285 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:14:08.574794 (+ 509us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:14:08.574876 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:14:08.574901 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:14:08.574987 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 15:14:08.575221 (+ 234us) server_negotiation.cc:299] Negotiation successful
0925 15:14:08.575551 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":769,"thread_start_us":573,"threads_started":1}
I20260925 15:14:08.579149 21642 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e1a68f3941314c55a1d8fbe688770692: Neither blocks nor log segments found. Creating new log.
I20260925 15:14:08.580090 21642 log.cc:824] T 00000000000000000000000000000000 P e1a68f3941314c55a1d8fbe688770692: Log is configured to *not* fsync() on all Append() calls
I20260925 15:14:08.582615 21642 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1a68f3941314c55a1d8fbe688770692: No bootstrap required, opened a new log
I20260925 15:14:08.589380 21642 raft_consensus.cc:348] T 00000000000000000000000000000000 P e1a68f3941314c55a1d8fbe688770692 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1a68f3941314c55a1d8fbe688770692" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 45349 } }
I20260925 15:14:08.589721 21642 raft_consensus.cc:374] T 00000000000000000000000000000000 P e1a68f3941314c55a1d8fbe688770692 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:14:08.589833 21642 raft_consensus.cc:729] T 00000000000000000000000000000000 P e1a68f3941314c55a1d8fbe688770692 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1a68f3941314c55a1d8fbe688770692, State: Initialized, Role: FOLLOWER
I20260925 15:14:08.590451 21642 consensus_queue.cc:260] T 00000000000000000000000000000000 P e1a68f3941314c55a1d8fbe688770692 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1a68f3941314c55a1d8fbe688770692" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 45349 } }
I20260925 15:14:08.590692 21642 raft_consensus.cc:388] T 00000000000000000000000000000000 P e1a68f3941314c55a1d8fbe688770692 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:14:08.590827 21642 raft_consensus.cc:482] T 00000000000000000000000000000000 P e1a68f3941314c55a1d8fbe688770692 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:14:08.591012 21642 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e1a68f3941314c55a1d8fbe688770692 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:14:08.593004 21642 raft_consensus.cc:504] T 00000000000000000000000000000000 P e1a68f3941314c55a1d8fbe688770692 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1a68f3941314c55a1d8fbe688770692" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 45349 } }
I20260925 15:14:08.593479 21642 leader_election.cc:302] T 00000000000000000000000000000000 P e1a68f3941314c55a1d8fbe688770692 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e1a68f3941314c55a1d8fbe688770692; no voters:
I20260925 15:14:08.594033 21642 leader_election.cc:288] T 00000000000000000000000000000000 P e1a68f3941314c55a1d8fbe688770692 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:14:08.594228 21647 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e1a68f3941314c55a1d8fbe688770692 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:14:08.594938 21647 raft_consensus.cc:686] T 00000000000000000000000000000000 P e1a68f3941314c55a1d8fbe688770692 [term 1 LEADER]: Becoming Leader. State: Replica: e1a68f3941314c55a1d8fbe688770692, State: Running, Role: LEADER
I20260925 15:14:08.595460 21647 consensus_queue.cc:237] T 00000000000000000000000000000000 P e1a68f3941314c55a1d8fbe688770692 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1a68f3941314c55a1d8fbe688770692" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 45349 } }
I20260925 15:14:08.596107 21642 sys_catalog.cc:564] T 00000000000000000000000000000000 P e1a68f3941314c55a1d8fbe688770692 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:14:08.598917 21649 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1a68f3941314c55a1d8fbe688770692 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1a68f3941314c55a1d8fbe688770692. Latest consensus state: current_term: 1 leader_uuid: "e1a68f3941314c55a1d8fbe688770692" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1a68f3941314c55a1d8fbe688770692" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 45349 } } }
I20260925 15:14:08.598896 21648 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1a68f3941314c55a1d8fbe688770692 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e1a68f3941314c55a1d8fbe688770692" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1a68f3941314c55a1d8fbe688770692" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 45349 } } }
I20260925 15:14:08.599313 21649 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1a68f3941314c55a1d8fbe688770692 [sys.catalog]: This master's current role is: LEADER
I20260925 15:14:08.599355 21648 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1a68f3941314c55a1d8fbe688770692 [sys.catalog]: This master's current role is: LEADER
I20260925 15:14:08.600558 21654 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:14:08.606914 21654 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:14:08.616264 21654 catalog_manager.cc:1347] Generated new cluster ID: df523b872a014cb99fd2d3abb9981689
I20260925 15:14:08.616374 21654 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:14:08.634965 21654 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:14:08.636847 21654 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:14:08.650240 21654 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e1a68f3941314c55a1d8fbe688770692: Generated new TSK 0
I20260925 15:14:08.651122 21654 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:14:08.658680 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349162370577-12320-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349162370577-12320-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349162370577-12320-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349162370577-12320-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.1:0
--local_ip_for_outbound_sockets=127.12.8.1
--webserver_interface=127.12.8.1
--webserver_port=0
--tserver_master_addrs=127.12.8.62:45349
--builtin_ntp_servers=127.12.8.20:33925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260925 15:14:09.000037 21666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:14:09.000339 21666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:14:09.000507 21666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:14:09.009287 21666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:14:09.009415 21666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:14:09.009611 21666 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.1
I20260925 15:14:09.020275 21666 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:33925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349162370577-12320-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349162370577-12320-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349162370577-12320-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.8.1
--webserver_port=0
--tserver_master_addrs=127.12.8.62:45349
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.1
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349162370577-12320-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:14:09.021591 21666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:14:09.023448 21666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:14:09.036665 21671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:14:09.036659 21672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:14:09.036674 21674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:14:09.037921 21666 server_base.cc:1034] running on GCE node
I20260925 15:14:09.038754 21666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:14:09.039942 21666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:14:09.041193 21666 hybrid_clock.cc:648] HybridClock initialized: now 1790349249041137 us; error 60 us; skew 500 ppm
I20260925 15:14:09.041689 21666 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:14:09.044909 21666 webserver.cc:460] Webserver started at http://127.12.8.1:37139/ using document root <none> and password file <none>
I20260925 15:14:09.046056 21666 fs_manager.cc:359] Metadata directory not provided
I20260925 15:14:09.046177 21666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:14:09.046609 21666 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:14:09.049403 21666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349162370577-12320-0/minicluster-data/ts-0/data/instance:
uuid: "a888d04758e6494980145705595e9d83"
format_stamp: "Formatted at 2026-09-25 15:14:09 on dist-test-slave-p468"
server_key: "65bdfacf09c469a329ddd4c0abeb8789"
server_key_iv: "f4cd4c246392462dd993f72ee1aef0fb"
server_key_version: "encryptionkey@0"
I20260925 15:14:09.050264 21666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349162370577-12320-0/minicluster-data/ts-0/wal/instance:
uuid: "a888d04758e6494980145705595e9d83"
format_stamp: "Formatted at 2026-09-25 15:14:09 on dist-test-slave-p468"
server_key: "65bdfacf09c469a329ddd4c0abeb8789"
server_key_iv: "f4cd4c246392462dd993f72ee1aef0fb"
server_key_version: "encryptionkey@0"
I20260925 15:14:09.055055 21666 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.002s
I20260925 15:14:09.058342 21680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:14:09.059584 21666 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 15:14:09.059849 21666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349162370577-12320-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349162370577-12320-0/minicluster-data/ts-0/wal
uuid: "a888d04758e6494980145705595e9d83"
format_stamp: "Formatted at 2026-09-25 15:14:09 on dist-test-slave-p468"
server_key: "65bdfacf09c469a329ddd4c0abeb8789"
server_key_iv: "f4cd4c246392462dd993f72ee1aef0fb"
server_key_version: "encryptionkey@0"
I20260925 15:14:09.060050 21666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349162370577-12320-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349162370577-12320-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349162370577-12320-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:14:09.077472 21666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:14:09.078399 21666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:14:09.078666 21666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:14:09.079556 21666 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:14:09.080984 21666 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:14:09.081115 21666 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:14:09.081239 21666 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:14:09.081326 21666 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:14:09.114990 21666 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.1:38649
I20260925 15:14:09.115039 21792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.1:38649 every 8 connection(s)
I20260925 15:14:09.116376 21666 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349162370577-12320-0/minicluster-data/ts-0/data/info.pb
I20260925 15:14:09.121249 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 21666
I20260925 15:14:09.121444 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349162370577-12320-0/minicluster-data/ts-0/wal/instance
I20260925 15:14:09.122059 12320 external_mini_cluster.cc:1428] Setting key 4f97d0e523ee438903f7feea81c1ada3
I20260925 15:14:09.124673 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349162370577-12320-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349162370577-12320-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349162370577-12320-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349162370577-12320-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.2:0
--local_ip_for_outbound_sockets=127.12.8.2
--webserver_interface=127.12.8.2
--webserver_port=0
--tserver_master_addrs=127.12.8.62:45349
--builtin_ntp_servers=127.12.8.20:33925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 15:14:09.128846 21795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:14:09.119161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.1:57293 (local address 127.12.8.62:45349)
0925 15:14:09.119603 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0925 15:14:09.119608 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:14:09.120679 (+ 1071us) server_negotiation.cc:407] Connection header received
0925 15:14:09.122614 (+ 1935us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:14:09.122619 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:14:09.122705 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:14:09.122823 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:14:09.123833 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:14:09.124561 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:14:09.126167 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:14:09.126597 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:14:09.126731 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:14:09.128045 (+ 1314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:14:09.128049 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:14:09.128052 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:14:09.128069 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:14:09.128110 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:14:09.128124 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:14:09.128223 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 15:14:09.128431 (+ 208us) server_negotiation.cc:299] Negotiation successful
0925 15:14:09.128517 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":248,"threads_started":1}
I20260925 15:14:09.129825 21796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:14:09.119653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:45349 (local address 127.12.8.1:57293)
0925 15:14:09.120521 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0925 15:14:09.120562 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 15:14:09.122342 (+ 1780us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:14:09.123046 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:14:09.123053 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:14:09.123145 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:14:09.123680 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:14:09.123695 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:14:09.124726 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:14:09.124731 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:14:09.126023 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:14:09.126039 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:14:09.126810 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:14:09.126818 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:14:09.127046 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:14:09.127748 (+ 702us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:14:09.127751 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:14:09.127788 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:14:09.127858 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:14:09.128220 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:14:09.128230 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:14:09.128241 (+ 11us) client_negotiation.cc:769] Sending connection context
0925 15:14:09.128419 (+ 178us) client_negotiation.cc:240] Negotiation successful
0925 15:14:09.128805 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":284,"threads_started":1}
I20260925 15:14:09.131410 21793 heartbeater.cc:347] Connected to a master server at 127.12.8.62:45349
I20260925 15:14:09.131701 21793 heartbeater.cc:464] Registering TS with master...
I20260925 15:14:09.132581 21793 heartbeater.cc:511] Master 127.12.8.62:45349 requested a full tablet report, sending...
I20260925 15:14:09.134935 21607 ts_manager.cc:194] Registered new tserver with Master: a888d04758e6494980145705595e9d83 (127.12.8.1:38649)
I20260925 15:14:09.137161 21607 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.8.1:57293
W20260925 15:14:09.447866 21797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:14:09.448139 21797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:14:09.448220 21797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:14:09.455265 21797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:14:09.455375 21797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:14:09.455480 21797 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.2
I20260925 15:14:09.464610 21797 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:33925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349162370577-12320-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349162370577-12320-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349162370577-12320-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.8.2
--webserver_port=0
--tserver_master_addrs=127.12.8.62:45349
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.2
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349162370577-12320-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:14:09.465867 21797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:14:09.467584 21797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:14:09.480660 21805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:14:09.480798 21802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:14:09.480820 21803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:14:09.481848 21797 server_base.cc:1034] running on GCE node
I20260925 15:14:09.482542 21797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:14:09.483530 21797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:14:09.484756 21797 hybrid_clock.cc:648] HybridClock initialized: now 1790349249484697 us; error 59 us; skew 500 ppm
I20260925 15:14:09.485109 21797 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:14:09.487382 21797 webserver.cc:460] Webserver started at http://127.12.8.2:37949/ using document root <none> and password file <none>
I20260925 15:14:09.488127 21797 fs_manager.cc:359] Metadata directory not provided
I20260925 15:14:09.488255 21797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:14:09.488564 21797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:14:09.493042 21797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349162370577-12320-0/minicluster-data/ts-1/data/instance:
uuid: "03e4e1f36fb548d1b013ac7f9eb04072"
format_stamp: "Formatted at 2026-09-25 15:14:09 on dist-test-slave-p468"
server_key: "e8506a3de5d795458a003a7aa98ebd44"
server_key_iv: "7d800e2a716a70a6c6df021f1d775c3c"
server_key_version: "encryptionkey@0"
I20260925 15:14:09.493860 21797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349162370577-12320-0/minicluster-data/ts-1/wal/instance:
uuid: "03e4e1f36fb548d1b013ac7f9eb04072"
format_stamp: "Formatted at 2026-09-25 15:14:09 on dist-test-slave-p468"
server_key: "e8506a3de5d795458a003a7aa98ebd44"
server_key_iv: "7d800e2a716a70a6c6df021f1d775c3c"
server_key_version: "encryptionkey@0"
I20260925 15:14:09.499300 21797 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260925 15:14:09.502960 21811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:14:09.504292 21797 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 15:14:09.504482 21797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349162370577-12320-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349162370577-12320-0/minicluster-data/ts-1/wal
uuid: "03e4e1f36fb548d1b013ac7f9eb04072"
format_stamp: "Formatted at 2026-09-25 15:14:09 on dist-test-slave-p468"
server_key: "e8506a3de5d795458a003a7aa98ebd44"
server_key_iv: "7d800e2a716a70a6c6df021f1d775c3c"
server_key_version: "encryptionkey@0"
I20260925 15:14:09.504719 21797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349162370577-12320-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349162370577-12320-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349162370577-12320-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:14:09.522166 21797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:14:09.523087 21797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:14:09.523355 21797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:14:09.524312 21797 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:14:09.525765 21797 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:14:09.525888 21797 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:14:09.526018 21797 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:14:09.526104 21797 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:14:09.559759 21797 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.2:38099
I20260925 15:14:09.559778 21923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.2:38099 every 8 connection(s)
I20260925 15:14:09.561041 21797 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349162370577-12320-0/minicluster-data/ts-1/data/info.pb
I20260925 15:14:09.569473 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 21797
I20260925 15:14:09.569669 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349162370577-12320-0/minicluster-data/ts-1/wal/instance
I20260925 15:14:09.570276 12320 external_mini_cluster.cc:1428] Setting key c27a4017cffdbf6fa02a105083a4976e
I20260925 15:14:09.573158 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349162370577-12320-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349162370577-12320-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349162370577-12320-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349162370577-12320-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.3:0
--local_ip_for_outbound_sockets=127.12.8.3
--webserver_interface=127.12.8.3
--webserver_port=0
--tserver_master_addrs=127.12.8.62:45349
--builtin_ntp_servers=127.12.8.20:33925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 15:14:09.575975 21795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:14:09.563790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.2:43483 (local address 127.12.8.62:45349)
0925 15:14:09.564003 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 15:14:09.564009 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:14:09.565647 (+ 1638us) server_negotiation.cc:407] Connection header received
0925 15:14:09.568055 (+ 2408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:14:09.568060 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:14:09.568132 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:14:09.568224 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:14:09.569588 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:14:09.570747 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:14:09.572719 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:14:09.573329 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:14:09.573446 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:14:09.575222 (+ 1776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:14:09.575229 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:14:09.575234 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:14:09.575254 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:14:09.575291 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:14:09.575305 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:14:09.575396 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 15:14:09.575653 (+ 257us) server_negotiation.cc:299] Negotiation successful
0925 15:14:09.575752 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 15:14:09.577399 21926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:14:09.564461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:45349 (local address 127.12.8.2:43483)
0925 15:14:09.565469 (+ 1008us) negotiation.cc:107] Waiting for socket to connect
0925 15:14:09.565509 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 15:14:09.567754 (+ 2245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:14:09.568490 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:14:09.568501 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:14:09.568622 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:14:09.569393 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:14:09.569417 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:14:09.570924 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:14:09.570929 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:14:09.572549 (+ 1620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:14:09.572572 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:14:09.573579 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:14:09.573588 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:14:09.573927 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:14:09.574844 (+ 917us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:14:09.574850 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:14:09.574903 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:14:09.575003 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:14:09.575417 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:14:09.575429 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:14:09.575440 (+ 11us) client_negotiation.cc:769] Sending connection context
0925 15:14:09.575622 (+ 182us) client_negotiation.cc:240] Negotiation successful
0925 15:14:09.576041 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":290,"threads_started":1}
I20260925 15:14:09.578945 21924 heartbeater.cc:347] Connected to a master server at 127.12.8.62:45349
I20260925 15:14:09.579313 21924 heartbeater.cc:464] Registering TS with master...
I20260925 15:14:09.580116 21924 heartbeater.cc:511] Master 127.12.8.62:45349 requested a full tablet report, sending...
I20260925 15:14:09.581744 21607 ts_manager.cc:194] Registered new tserver with Master: 03e4e1f36fb548d1b013ac7f9eb04072 (127.12.8.2:38099)
I20260925 15:14:09.582677 21607 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.8.2:43483
W20260925 15:14:09.901875 21927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:14:09.902262 21927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:14:09.902418 21927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:14:09.909897 21927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:14:09.910032 21927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:14:09.910259 21927 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.3
I20260925 15:14:09.919725 21927 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:33925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349162370577-12320-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349162370577-12320-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349162370577-12320-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.8.3
--webserver_port=0
--tserver_master_addrs=127.12.8.62:45349
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.3
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349162370577-12320-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:14:09.921056 21927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:14:09.922830 21927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:14:09.935813 21935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:14:09.935786 21932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:14:09.935877 21933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:14:09.936630 21927 server_base.cc:1034] running on GCE node
I20260925 15:14:09.937206 21927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:14:09.938143 21927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:14:09.939388 21927 hybrid_clock.cc:648] HybridClock initialized: now 1790349249939337 us; error 52 us; skew 500 ppm
I20260925 15:14:09.939697 21927 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:14:09.942004 21927 webserver.cc:460] Webserver started at http://127.12.8.3:44791/ using document root <none> and password file <none>
I20260925 15:14:09.942759 21927 fs_manager.cc:359] Metadata directory not provided
I20260925 15:14:09.942893 21927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:14:09.943221 21927 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:14:09.946107 21927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349162370577-12320-0/minicluster-data/ts-2/data/instance:
uuid: "90047fc15dbd4653a6ef4a8de937e197"
format_stamp: "Formatted at 2026-09-25 15:14:09 on dist-test-slave-p468"
server_key: "f9e4bd58e09649c027a0f053b45471d2"
server_key_iv: "466f2861dcba3a5bd4f35e6ef0c47d05"
server_key_version: "encryptionkey@0"
I20260925 15:14:09.946985 21927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349162370577-12320-0/minicluster-data/ts-2/wal/instance:
uuid: "90047fc15dbd4653a6ef4a8de937e197"
format_stamp: "Formatted at 2026-09-25 15:14:09 on dist-test-slave-p468"
server_key: "f9e4bd58e09649c027a0f053b45471d2"
server_key_iv: "466f2861dcba3a5bd4f35e6ef0c47d05"
server_key_version: "encryptionkey@0"
I20260925 15:14:09.952065 21927 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 15:14:09.955431 21941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:14:09.956692 21927 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 15:14:09.956867 21927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349162370577-12320-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349162370577-12320-0/minicluster-data/ts-2/wal
uuid: "90047fc15dbd4653a6ef4a8de937e197"
format_stamp: "Formatted at 2026-09-25 15:14:09 on dist-test-slave-p468"
server_key: "f9e4bd58e09649c027a0f053b45471d2"
server_key_iv: "466f2861dcba3a5bd4f35e6ef0c47d05"
server_key_version: "encryptionkey@0"
I20260925 15:14:09.957082 21927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349162370577-12320-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349162370577-12320-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349162370577-12320-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:14:09.976055 21927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:14:09.976918 21927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:14:09.977201 21927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:14:09.978171 21927 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:14:09.979686 21927 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:14:09.979810 21927 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:14:09.979940 21927 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:14:09.980027 21927 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:14:10.013447 21927 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.3:45019
I20260925 15:14:10.013549 22053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.3:45019 every 8 connection(s)
I20260925 15:14:10.015007 21927 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349162370577-12320-0/minicluster-data/ts-2/data/info.pb
I20260925 15:14:10.017969 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 21927
I20260925 15:14:10.018124 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349162370577-12320-0/minicluster-data/ts-2/wal/instance
I20260925 15:14:10.018711 12320 external_mini_cluster.cc:1428] Setting key d3ce9772cabc63ea0d8ada799e7e5bf8
I20260925 15:14:10.028062 21795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:14:10.017682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.3:38835 (local address 127.12.8.62:45349)
0925 15:14:10.017857 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 15:14:10.017861 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:14:10.019762 (+ 1901us) server_negotiation.cc:407] Connection header received
0925 15:14:10.021725 (+ 1963us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:14:10.021730 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:14:10.021821 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:14:10.021928 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:14:10.023106 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:14:10.023823 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:14:10.025482 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:14:10.025889 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:14:10.025981 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:14:10.027368 (+ 1387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:14:10.027373 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:14:10.027376 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:14:10.027393 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:14:10.027424 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:14:10.027436 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:14:10.027508 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 15:14:10.027753 (+ 245us) server_negotiation.cc:299] Negotiation successful
0925 15:14:10.027825 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 15:14:10.029071 22056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:14:10.018414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:45349 (local address 127.12.8.3:38835)
0925 15:14:10.019614 (+ 1200us) negotiation.cc:107] Waiting for socket to connect
0925 15:14:10.019651 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 15:14:10.021400 (+ 1749us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:14:10.022181 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:14:10.022253 (+ 72us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:14:10.022362 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:14:10.022958 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:14:10.022975 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:14:10.023965 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:14:10.023969 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:14:10.025359 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:14:10.025371 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:14:10.026095 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:14:10.026101 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:14:10.026382 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:14:10.027045 (+ 663us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:14:10.027048 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:14:10.027098 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:14:10.027186 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:14:10.027553 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:14:10.027562 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:14:10.027569 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 15:14:10.027716 (+ 147us) client_negotiation.cc:240] Negotiation successful
0925 15:14:10.028004 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"thread_start_us":398,"threads_started":1}
I20260925 15:14:10.030404 22054 heartbeater.cc:347] Connected to a master server at 127.12.8.62:45349
I20260925 15:14:10.030716 22054 heartbeater.cc:464] Registering TS with master...
I20260925 15:14:10.031477 22054 heartbeater.cc:511] Master 127.12.8.62:45349 requested a full tablet report, sending...
I20260925 15:14:10.032974 21607 ts_manager.cc:194] Registered new tserver with Master: 90047fc15dbd4653a6ef4a8de937e197 (127.12.8.3:45019)
I20260925 15:14:10.034046 21607 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.8.3:38835
I20260925 15:14:10.034166 12320 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 15:14:10.040367 21795 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:14:10.038834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55182 (local address 127.12.8.62:45349)
0925 15:14:10.038997 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 15:14:10.039001 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:14:10.039485 (+ 484us) server_negotiation.cc:407] Connection header received
0925 15:14:10.039628 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:14:10.039632 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:14:10.039701 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:14:10.039784 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:14:10.040109 (+ 325us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55182: BlockingRecv error: recv got EOF from 127.0.0.1:55182 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 15:14:10.041153 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 21666
I20260925 15:14:10.048636 22063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:14:10.042087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38674 (local address 127.12.8.1:38649)
0925 15:14:10.042691 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0925 15:14:10.042697 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:14:10.042724 (+ 27us) server_negotiation.cc:407] Connection header received
0925 15:14:10.042779 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:14:10.042785 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:14:10.042986 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:14:10.043124 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:14:10.043459 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:14:10.044454 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:14:10.045628 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:14:10.046241 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:14:10.046353 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:14:10.047354 (+ 1001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:14:10.047362 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:14:10.047373 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:14:10.047929 (+ 556us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:14:10.047970 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:14:10.047987 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:14:10.048105 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 15:14:10.048159 (+ 54us) server_negotiation.cc:299] Negotiation successful
0925 15:14:10.048298 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":323,"threads_started":1}
I20260925 15:14:10.049731 21788 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:14:10.131633 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 21666
I20260925 15:14:10.147389 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 21797
I20260925 15:14:10.155256 22065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:14:10.148381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34318 (local address 127.12.8.2:38099)
0925 15:14:10.148948 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0925 15:14:10.148955 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:14:10.148981 (+ 26us) server_negotiation.cc:407] Connection header received
0925 15:14:10.149048 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:14:10.149054 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:14:10.149268 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:14:10.149401 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:14:10.149820 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:14:10.150943 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:14:10.152166 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:14:10.152705 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:14:10.152802 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:14:10.153805 (+ 1003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:14:10.153813 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:14:10.153828 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:14:10.154510 (+ 682us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:14:10.154556 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:14:10.154576 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:14:10.154679 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 15:14:10.154810 (+ 131us) server_negotiation.cc:299] Negotiation successful
0925 15:14:10.154930 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":303,"threads_started":1}
I20260925 15:14:10.156499 21919 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:14:10.238468 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 21797
I20260925 15:14:10.254036 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 21927
I20260925 15:14:10.262826 22067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:14:10.255340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59852 (local address 127.12.8.3:45019)
0925 15:14:10.256083 (+ 743us) server_negotiation.cc:206] Beginning negotiation
0925 15:14:10.256132 (+ 49us) server_negotiation.cc:399] Waiting for connection header
0925 15:14:10.256205 (+ 73us) server_negotiation.cc:407] Connection header received
0925 15:14:10.256289 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:14:10.256298 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:14:10.256544 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:14:10.256680 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:14:10.257160 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:14:10.258252 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:14:10.259468 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:14:10.260018 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:14:10.260154 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:14:10.261309 (+ 1155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:14:10.261317 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:14:10.261330 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:14:10.262011 (+ 681us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:14:10.262063 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:14:10.262082 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:14:10.262180 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 15:14:10.262384 (+ 204us) server_negotiation.cc:299] Negotiation successful
0925 15:14:10.262519 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":418,"threads_started":1}
I20260925 15:14:10.264058 22049 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:14:10.348738 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 21927
I20260925 15:14:10.364385 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 21576
I20260925 15:14:10.365672 21637 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:14:10.467124 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 21576
2026-09-25T15:14:10Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2392 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-25T15:14:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:14:10Z Disabled control of system clock
I20260925 15:14:10.536847 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349162370577-12320-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349162370577-12320-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349162370577-12320-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.62:38959
--webserver_interface=127.12.8.62
--webserver_port=0
--builtin_ntp_servers=127.12.8.20:40315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.62:38959
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 15:14:10.855705 22073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:14:10.855981 22073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:14:10.856159 22073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:14:10.863550 22073 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:14:10.863667 22073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:14:10.863780 22073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:14:10.863893 22073 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:14:10.863970 22073 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:14:10.873086 22073 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:40315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349162370577-12320-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349162370577-12320-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.8.62:38959
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.62:38959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.8.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349162370577-12320-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:14:10.874512 22073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:14:10.876394 22073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:14:10.888603 22081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:14:10.888656 22078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:14:10.888684 22079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:14:10.889578 22073 server_base.cc:1034] running on GCE node
I20260925 15:14:10.890479 22073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:14:10.891919 22073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:14:10.893147 22073 hybrid_clock.cc:648] HybridClock initialized: now 1790349250893114 us; error 31 us; skew 500 ppm
I20260925 15:14:10.893505 22073 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:14:10.895865 22073 webserver.cc:460] Webserver started at http://127.12.8.62:39773/ using document root <none> and password file <none>
I20260925 15:14:10.896575 22073 fs_manager.cc:359] Metadata directory not provided
I20260925 15:14:10.896687 22073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:14:10.897017 22073 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:14:10.899879 22073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349162370577-12320-0/minicluster-data/master-0/data/instance:
uuid: "2abfaf4005de44c18cd627d7b606714e"
format_stamp: "Formatted at 2026-09-25 15:14:10 on dist-test-slave-p468"
server_key: "1ca07704d154ac5b156bca59009798bd"
server_key_iv: "6c657ec5a170ce2707e221285831b7e8"
server_key_version: "encryptionkey@0"
I20260925 15:14:10.900668 22073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349162370577-12320-0/minicluster-data/master-0/wal/instance:
uuid: "2abfaf4005de44c18cd627d7b606714e"
format_stamp: "Formatted at 2026-09-25 15:14:10 on dist-test-slave-p468"
server_key: "1ca07704d154ac5b156bca59009798bd"
server_key_iv: "6c657ec5a170ce2707e221285831b7e8"
server_key_version: "encryptionkey@0"
I20260925 15:14:10.905610 22073 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260925 15:14:10.909052 22087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:14:10.910406 22073 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:14:10.910579 22073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349162370577-12320-0/minicluster-data/master-0/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349162370577-12320-0/minicluster-data/master-0/wal
uuid: "2abfaf4005de44c18cd627d7b606714e"
format_stamp: "Formatted at 2026-09-25 15:14:10 on dist-test-slave-p468"
server_key: "1ca07704d154ac5b156bca59009798bd"
server_key_iv: "6c657ec5a170ce2707e221285831b7e8"
server_key_version: "encryptionkey@0"
I20260925 15:14:10.910808 22073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349162370577-12320-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349162370577-12320-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349162370577-12320-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:14:10.916715 22073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:14:10.917478 22073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:14:10.917788 22073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:14:10.935741 22073 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.62:38959
I20260925 15:14:10.935767 22138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.62:38959 every 8 connection(s)
I20260925 15:14:10.937120 22073 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
I20260925 15:14:10.940817 22139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:14:10.946014 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 22073
I20260925 15:14:10.946295 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349162370577-12320-0/minicluster-data/master-0/wal/instance
I20260925 15:14:10.946900 12320 external_mini_cluster.cc:1428] Setting key 368a5d2efb7e86713f41e0732abdb297
I20260925 15:14:10.951436 22139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2abfaf4005de44c18cd627d7b606714e: Bootstrap starting.
I20260925 15:14:10.954979 22139 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2abfaf4005de44c18cd627d7b606714e: Neither blocks nor log segments found. Creating new log.
I20260925 15:14:10.955327 22142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:14:10.949077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43506 (local address 127.12.8.62:38959)
0925 15:14:10.950102 (+ 1025us) server_negotiation.cc:206] Beginning negotiation
0925 15:14:10.950113 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:14:10.950154 (+ 41us) server_negotiation.cc:407] Connection header received
0925 15:14:10.951831 (+ 1677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:14:10.951853 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:14:10.952246 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:14:10.952568 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:14:10.953686 (+ 1118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:14:10.953691 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:14:10.953703 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:14:10.953716 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:14:10.953777 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:14:10.953793 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:14:10.953949 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 15:14:10.954084 (+ 135us) server_negotiation.cc:299] Negotiation successful
0925 15:14:10.954392 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":402,"threads_started":1}
I20260925 15:14:10.956112 22139 log.cc:824] T 00000000000000000000000000000000 P 2abfaf4005de44c18cd627d7b606714e: Log is configured to *not* fsync() on all Append() calls
I20260925 15:14:10.958931 22139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2abfaf4005de44c18cd627d7b606714e: No bootstrap required, opened a new log
I20260925 15:14:10.966091 22139 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2abfaf4005de44c18cd627d7b606714e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2abfaf4005de44c18cd627d7b606714e" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 38959 } }
I20260925 15:14:10.966531 22139 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2abfaf4005de44c18cd627d7b606714e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:14:10.966637 22139 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2abfaf4005de44c18cd627d7b606714e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2abfaf4005de44c18cd627d7b606714e, State: Initialized, Role: FOLLOWER
I20260925 15:14:10.967221 22139 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2abfaf4005de44c18cd627d7b606714e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2abfaf4005de44c18cd627d7b606714e" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 38959 } }
I20260925 15:14:10.967458 22139 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2abfaf4005de44c18cd627d7b606714e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:14:10.967605 22139 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2abfaf4005de44c18cd627d7b606714e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:14:10.967787 22139 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2abfaf4005de44c18cd627d7b606714e [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:14:10.969924 22139 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2abfaf4005de44c18cd627d7b606714e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2abfaf4005de44c18cd627d7b606714e" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 38959 } }
I20260925 15:14:10.970464 22139 leader_election.cc:302] T 00000000000000000000000000000000 P 2abfaf4005de44c18cd627d7b606714e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2abfaf4005de44c18cd627d7b606714e; no voters:
I20260925 15:14:10.971010 22139 leader_election.cc:288] T 00000000000000000000000000000000 P 2abfaf4005de44c18cd627d7b606714e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:14:10.971138 22144 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2abfaf4005de44c18cd627d7b606714e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:14:10.971894 22144 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2abfaf4005de44c18cd627d7b606714e [term 1 LEADER]: Becoming Leader. State: Replica: 2abfaf4005de44c18cd627d7b606714e, State: Running, Role: LEADER
I20260925 15:14:10.972438 22144 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2abfaf4005de44c18cd627d7b606714e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2abfaf4005de44c18cd627d7b606714e" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 38959 } }
I20260925 15:14:10.973044 22139 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2abfaf4005de44c18cd627d7b606714e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:14:10.975586 22145 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2abfaf4005de44c18cd627d7b606714e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2abfaf4005de44c18cd627d7b606714e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2abfaf4005de44c18cd627d7b606714e" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 38959 } } }
I20260925 15:14:10.976025 22145 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2abfaf4005de44c18cd627d7b606714e [sys.catalog]: This master's current role is: LEADER
I20260925 15:14:10.976601 22146 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2abfaf4005de44c18cd627d7b606714e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2abfaf4005de44c18cd627d7b606714e. Latest consensus state: current_term: 1 leader_uuid: "2abfaf4005de44c18cd627d7b606714e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2abfaf4005de44c18cd627d7b606714e" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 38959 } } }
I20260925 15:14:10.976970 22146 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2abfaf4005de44c18cd627d7b606714e [sys.catalog]: This master's current role is: LEADER
W20260925 15:14:10.983711 22160 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2abfaf4005de44c18cd627d7b606714e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:14:10.983863 22160 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:14:10.986735 22153 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:14:10.988843 22153 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:14:10.997640 22153 catalog_manager.cc:1347] Generated new cluster ID: d27d254e668b43b183228afd066142d7
I20260925 15:14:10.997769 22153 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:14:11.029232 22153 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:14:11.029690 22153 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:14:11.039702 22153 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2abfaf4005de44c18cd627d7b606714e: Generated new TSK 0
I20260925 15:14:11.040544 22153 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:14:11.052721 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349162370577-12320-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349162370577-12320-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349162370577-12320-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349162370577-12320-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.1:0
--local_ip_for_outbound_sockets=127.12.8.1
--webserver_interface=127.12.8.1
--webserver_port=0
--tserver_master_addrs=127.12.8.62:38959
--builtin_ntp_servers=127.12.8.20:40315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 15:14:11.398102 22163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:14:11.398380 22163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:14:11.398528 22163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:14:11.405625 22163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:14:11.405709 22163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:14:11.405912 22163 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.1
I20260925 15:14:11.414955 22163 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:40315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349162370577-12320-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349162370577-12320-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349162370577-12320-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.8.1
--webserver_port=0
--tserver_master_addrs=127.12.8.62:38959
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.1
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349162370577-12320-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:14:11.416270 22163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:14:11.418044 22163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:14:11.430828 22171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:14:11.430966 22168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:14:11.431012 22169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:14:11.435588 22163 server_base.cc:1034] running on GCE node
I20260925 15:14:11.436211 22163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:14:11.437117 22163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:14:11.438285 22163 hybrid_clock.cc:648] HybridClock initialized: now 1790349251438250 us; error 38 us; skew 500 ppm
I20260925 15:14:11.438589 22163 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:14:11.440697 22163 webserver.cc:460] Webserver started at http://127.12.8.1:41411/ using document root <none> and password file <none>
I20260925 15:14:11.441344 22163 fs_manager.cc:359] Metadata directory not provided
I20260925 15:14:11.441452 22163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:14:11.441763 22163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:14:11.444569 22163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349162370577-12320-0/minicluster-data/ts-0/data/instance:
uuid: "f21d571c89ab40bdb1dd984f499fa362"
format_stamp: "Formatted at 2026-09-25 15:14:11 on dist-test-slave-p468"
server_key: "3ca2c72b35e2bb96e3b4650d7dbb4492"
server_key_iv: "f35baf97ca7880d88d3a86997bbf134f"
server_key_version: "encryptionkey@0"
I20260925 15:14:11.445329 22163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349162370577-12320-0/minicluster-data/ts-0/wal/instance:
uuid: "f21d571c89ab40bdb1dd984f499fa362"
format_stamp: "Formatted at 2026-09-25 15:14:11 on dist-test-slave-p468"
server_key: "3ca2c72b35e2bb96e3b4650d7dbb4492"
server_key_iv: "f35baf97ca7880d88d3a86997bbf134f"
server_key_version: "encryptionkey@0"
I20260925 15:14:11.450345 22163 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 15:14:11.453877 22177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:14:11.455185 22163 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:14:11.455356 22163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349162370577-12320-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349162370577-12320-0/minicluster-data/ts-0/wal
uuid: "f21d571c89ab40bdb1dd984f499fa362"
format_stamp: "Formatted at 2026-09-25 15:14:11 on dist-test-slave-p468"
server_key: "3ca2c72b35e2bb96e3b4650d7dbb4492"
server_key_iv: "f35baf97ca7880d88d3a86997bbf134f"
server_key_version: "encryptionkey@0"
I20260925 15:14:11.455557 22163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349162370577-12320-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349162370577-12320-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349162370577-12320-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:14:11.460906 22163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:14:11.461596 22163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:14:11.461812 22163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:14:11.462668 22163 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:14:11.463904 22163 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:14:11.463987 22163 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:14:11.464063 22163 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:14:11.464116 22163 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:14:11.496729 22163 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.1:39075
I20260925 15:14:11.496757 22289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.1:39075 every 8 connection(s)
I20260925 15:14:11.498045 22163 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349162370577-12320-0/minicluster-data/ts-0/data/info.pb
I20260925 15:14:11.506222 22292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:14:11.500623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.1:55259 (local address 127.12.8.62:38959)
0925 15:14:11.501057 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0925 15:14:11.501062 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:14:11.502295 (+ 1233us) server_negotiation.cc:407] Connection header received
0925 15:14:11.503937 (+ 1642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:14:11.503943 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:14:11.504024 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:14:11.504141 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:14:11.505380 (+ 1239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:14:11.505386 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:14:11.505390 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:14:11.505407 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:14:11.505454 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:14:11.505465 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:14:11.505585 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0925 15:14:11.505793 (+ 208us) server_negotiation.cc:299] Negotiation successful
0925 15:14:11.505880 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":233,"threads_started":1}
I20260925 15:14:11.507120 22293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:14:11.501232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:38959 (local address 127.12.8.1:55259)
0925 15:14:11.502108 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0925 15:14:11.502144 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 15:14:11.503670 (+ 1526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:14:11.504377 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:14:11.504384 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:14:11.504474 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:14:11.505102 (+ 628us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:14:11.505105 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:14:11.505140 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:14:11.505222 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:14:11.505557 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:14:11.505568 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:14:11.505575 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 15:14:11.505896 (+ 321us) client_negotiation.cc:240] Negotiation successful
0925 15:14:11.506144 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":249,"threads_started":1}
I20260925 15:14:11.507432 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 22163
I20260925 15:14:11.507681 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349162370577-12320-0/minicluster-data/ts-0/wal/instance
I20260925 15:14:11.508411 12320 external_mini_cluster.cc:1428] Setting key 1688ed011fc891bcc99e4f2757916eb8
I20260925 15:14:11.508611 22290 heartbeater.cc:347] Connected to a master server at 127.12.8.62:38959
I20260925 15:14:11.508894 22290 heartbeater.cc:464] Registering TS with master...
I20260925 15:14:11.509573 22290 heartbeater.cc:511] Master 127.12.8.62:38959 requested a full tablet report, sending...
I20260925 15:14:11.511176 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349162370577-12320-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349162370577-12320-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349162370577-12320-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349162370577-12320-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.2:0
--local_ip_for_outbound_sockets=127.12.8.2
--webserver_interface=127.12.8.2
--webserver_port=0
--tserver_master_addrs=127.12.8.62:38959
--builtin_ntp_servers=127.12.8.20:40315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260925 15:14:11.511646 22104 ts_manager.cc:194] Registered new tserver with Master: f21d571c89ab40bdb1dd984f499fa362 (127.12.8.1:39075)
W20260925 15:14:11.828680 22294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:14:11.828979 22294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:14:11.829139 22294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:14:11.836190 22294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:14:11.836314 22294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:14:11.836493 22294 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.2
I20260925 15:14:11.845181 22294 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:40315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349162370577-12320-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349162370577-12320-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349162370577-12320-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.8.2
--webserver_port=0
--tserver_master_addrs=127.12.8.62:38959
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.2
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349162370577-12320-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:14:11.846540 22294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:14:11.848274 22294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:14:11.861912 22299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:14:11.862051 22300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:14:11.862772 22302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:14:11.864751 22294 server_base.cc:1034] running on GCE node
I20260925 15:14:11.865450 22294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:14:11.866644 22294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:14:11.867856 22294 hybrid_clock.cc:648] HybridClock initialized: now 1790349251867806 us; error 51 us; skew 500 ppm
I20260925 15:14:11.868156 22294 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:14:11.870326 22294 webserver.cc:460] Webserver started at http://127.12.8.2:35599/ using document root <none> and password file <none>
I20260925 15:14:11.870990 22294 fs_manager.cc:359] Metadata directory not provided
I20260925 15:14:11.871074 22294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:14:11.871404 22294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:14:11.874382 22294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349162370577-12320-0/minicluster-data/ts-1/data/instance:
uuid: "8c91e45abea54d94aadb4ddd4ef0eed6"
format_stamp: "Formatted at 2026-09-25 15:14:11 on dist-test-slave-p468"
server_key: "8f4517fdaf33361650da244783fd6f8f"
server_key_iv: "bbaf6568a032b89c6b6ebd5e83542c87"
server_key_version: "encryptionkey@0"
I20260925 15:14:11.875175 22294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349162370577-12320-0/minicluster-data/ts-1/wal/instance:
uuid: "8c91e45abea54d94aadb4ddd4ef0eed6"
format_stamp: "Formatted at 2026-09-25 15:14:11 on dist-test-slave-p468"
server_key: "8f4517fdaf33361650da244783fd6f8f"
server_key_iv: "bbaf6568a032b89c6b6ebd5e83542c87"
server_key_version: "encryptionkey@0"
I20260925 15:14:11.880128 22294 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 15:14:11.883697 22308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:14:11.885072 22294 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 15:14:11.885250 22294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349162370577-12320-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349162370577-12320-0/minicluster-data/ts-1/wal
uuid: "8c91e45abea54d94aadb4ddd4ef0eed6"
format_stamp: "Formatted at 2026-09-25 15:14:11 on dist-test-slave-p468"
server_key: "8f4517fdaf33361650da244783fd6f8f"
server_key_iv: "bbaf6568a032b89c6b6ebd5e83542c87"
server_key_version: "encryptionkey@0"
I20260925 15:14:11.885463 22294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349162370577-12320-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349162370577-12320-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349162370577-12320-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:14:11.891170 22294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:14:11.892015 22294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:14:11.892297 22294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:14:11.893280 22294 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:14:11.894802 22294 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:14:11.894929 22294 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:14:11.895054 22294 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:14:11.895143 22294 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:14:11.927974 22294 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.2:46623
I20260925 15:14:11.928068 22420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.2:46623 every 8 connection(s)
I20260925 15:14:11.929273 22294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349162370577-12320-0/minicluster-data/ts-1/data/info.pb
I20260925 15:14:11.931671 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 22294
I20260925 15:14:11.931866 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349162370577-12320-0/minicluster-data/ts-1/wal/instance
I20260925 15:14:11.932402 12320 external_mini_cluster.cc:1428] Setting key a56f3dd785191c3c7af00e6da9d745a5
I20260925 15:14:11.935515 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349162370577-12320-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349162370577-12320-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349162370577-12320-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349162370577-12320-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.3:0
--local_ip_for_outbound_sockets=127.12.8.3
--webserver_interface=127.12.8.3
--webserver_port=0
--tserver_master_addrs=127.12.8.62:38959
--builtin_ntp_servers=127.12.8.20:40315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260925 15:14:11.938876 22292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:14:11.932036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.2:47189 (local address 127.12.8.62:38959)
0925 15:14:11.932227 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 15:14:11.932232 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:14:11.933962 (+ 1730us) server_negotiation.cc:407] Connection header received
0925 15:14:11.936307 (+ 2345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:14:11.936315 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:14:11.936385 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:14:11.936504 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:14:11.938029 (+ 1525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:14:11.938038 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:14:11.938045 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:14:11.938067 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:14:11.938109 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:14:11.938126 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:14:11.938295 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0925 15:14:11.938482 (+ 187us) server_negotiation.cc:299] Negotiation successful
0925 15:14:11.938546 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 15:14:11.939885 22423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:14:11.932664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:38959 (local address 127.12.8.2:47189)
0925 15:14:11.933791 (+ 1127us) negotiation.cc:107] Waiting for socket to connect
0925 15:14:11.933846 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 15:14:11.935968 (+ 2122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:14:11.936765 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:14:11.936777 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:14:11.936898 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:14:11.937688 (+ 790us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:14:11.937693 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:14:11.937739 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:14:11.937842 (+ 103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:14:11.938259 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:14:11.938269 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:14:11.938276 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 15:14:11.938550 (+ 274us) client_negotiation.cc:240] Negotiation successful
0925 15:14:11.938810 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":334,"threads_started":1}
I20260925 15:14:11.941219 22421 heartbeater.cc:347] Connected to a master server at 127.12.8.62:38959
I20260925 15:14:11.941534 22421 heartbeater.cc:464] Registering TS with master...
I20260925 15:14:11.942245 22421 heartbeater.cc:511] Master 127.12.8.62:38959 requested a full tablet report, sending...
I20260925 15:14:11.943632 22104 ts_manager.cc:194] Registered new tserver with Master: 8c91e45abea54d94aadb4ddd4ef0eed6 (127.12.8.2:46623)
W20260925 15:14:12.259330 22424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:14:12.259620 22424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:14:12.259774 22424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:14:12.267241 22424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:14:12.267352 22424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:14:12.267565 22424 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.3
I20260925 15:14:12.277132 22424 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:40315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349162370577-12320-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349162370577-12320-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349162370577-12320-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.8.3
--webserver_port=0
--tserver_master_addrs=127.12.8.62:38959
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.3
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349162370577-12320-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:14:12.278563 22424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:14:12.280372 22424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:14:12.293464 22429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:14:12.293546 22430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:14:12.293576 22432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:14:12.294468 22424 server_base.cc:1034] running on GCE node
I20260925 15:14:12.295090 22424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:14:12.296027 22424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:14:12.297276 22424 hybrid_clock.cc:648] HybridClock initialized: now 1790349252297202 us; error 54 us; skew 500 ppm
I20260925 15:14:12.297674 22424 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:14:12.300029 22424 webserver.cc:460] Webserver started at http://127.12.8.3:45267/ using document root <none> and password file <none>
I20260925 15:14:12.300745 22424 fs_manager.cc:359] Metadata directory not provided
I20260925 15:14:12.300885 22424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:14:12.301216 22424 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:14:12.304206 22424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349162370577-12320-0/minicluster-data/ts-2/data/instance:
uuid: "ef3efd4aa23d4249abcce1fce0f5cb31"
format_stamp: "Formatted at 2026-09-25 15:14:12 on dist-test-slave-p468"
server_key: "fffeb4499407f865d70b05d06a5681b1"
server_key_iv: "706c5b87929df687bdd752da9e91bd7a"
server_key_version: "encryptionkey@0"
I20260925 15:14:12.305035 22424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349162370577-12320-0/minicluster-data/ts-2/wal/instance:
uuid: "ef3efd4aa23d4249abcce1fce0f5cb31"
format_stamp: "Formatted at 2026-09-25 15:14:12 on dist-test-slave-p468"
server_key: "fffeb4499407f865d70b05d06a5681b1"
server_key_iv: "706c5b87929df687bdd752da9e91bd7a"
server_key_version: "encryptionkey@0"
I20260925 15:14:12.310245 22424 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260925 15:14:12.314524 22438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:14:12.316071 22424 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 15:14:12.316246 22424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349162370577-12320-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349162370577-12320-0/minicluster-data/ts-2/wal
uuid: "ef3efd4aa23d4249abcce1fce0f5cb31"
format_stamp: "Formatted at 2026-09-25 15:14:12 on dist-test-slave-p468"
server_key: "fffeb4499407f865d70b05d06a5681b1"
server_key_iv: "706c5b87929df687bdd752da9e91bd7a"
server_key_version: "encryptionkey@0"
I20260925 15:14:12.316469 22424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349162370577-12320-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349162370577-12320-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349162370577-12320-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:14:12.322548 22424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:14:12.323367 22424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:14:12.323635 22424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:14:12.324582 22424 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:14:12.325951 22424 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:14:12.326037 22424 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:14:12.326177 22424 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:14:12.326270 22424 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260925 15:14:12.358793 22424 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.3:37513
I20260925 15:14:12.358901 22550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.3:37513 every 8 connection(s)
I20260925 15:14:12.360217 22424 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349162370577-12320-0/minicluster-data/ts-2/data/info.pb
I20260925 15:14:12.361169 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 22424
I20260925 15:14:12.361379 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349162370577-12320-0/minicluster-data/ts-2/wal/instance
I20260925 15:14:12.361966 12320 external_mini_cluster.cc:1428] Setting key d5d49e63be2dd24ffd212ffa407cab9b
I20260925 15:14:12.369557 22292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:14:12.363445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.3:36165 (local address 127.12.8.62:38959)
0925 15:14:12.363676 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0925 15:14:12.363686 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:14:12.365500 (+ 1814us) server_negotiation.cc:407] Connection header received
0925 15:14:12.367486 (+ 1986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:14:12.367490 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:14:12.367539 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:14:12.367622 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:14:12.368900 (+ 1278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:14:12.368904 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:14:12.368907 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:14:12.368921 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:14:12.368951 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:14:12.368961 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:14:12.369056 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 15:14:12.369309 (+ 253us) server_negotiation.cc:299] Negotiation successful
0925 15:14:12.369358 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 15:14:12.370601 22553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:14:12.364150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:38959 (local address 127.12.8.3:36165)
0925 15:14:12.365339 (+ 1189us) negotiation.cc:107] Waiting for socket to connect
0925 15:14:12.365387 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 15:14:12.367231 (+ 1844us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:14:12.367832 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:14:12.367840 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:14:12.367932 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:14:12.368617 (+ 685us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:14:12.368621 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:14:12.368658 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:14:12.368746 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:14:12.369127 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:14:12.369134 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:14:12.369140 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 15:14:12.369361 (+ 221us) client_negotiation.cc:240] Negotiation successful
0925 15:14:12.369556 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":369,"threads_started":1}
I20260925 15:14:12.371961 22551 heartbeater.cc:347] Connected to a master server at 127.12.8.62:38959
I20260925 15:14:12.372255 22551 heartbeater.cc:464] Registering TS with master...
I20260925 15:14:12.372856 22551 heartbeater.cc:511] Master 127.12.8.62:38959 requested a full tablet report, sending...
I20260925 15:14:12.374250 22104 ts_manager.cc:194] Registered new tserver with Master: ef3efd4aa23d4249abcce1fce0f5cb31 (127.12.8.3:37513)
I20260925 15:14:12.378572 12320 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 15:14:12.384529 22292 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:14:12.382837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43514 (local address 127.12.8.62:38959)
0925 15:14:12.383005 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 15:14:12.383018 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 15:14:12.383565 (+ 547us) server_negotiation.cc:407] Connection header received
0925 15:14:12.383766 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:14:12.383772 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:14:12.383831 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:14:12.383932 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:14:12.384275 (+ 343us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43514: BlockingRecv error: recv got EOF from 127.0.0.1:43514 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 15:14:12.385507 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 22163
I20260925 15:14:12.389241 22560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:14:12.386500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48524 (local address 127.12.8.1:39075)
0925 15:14:12.387108 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0925 15:14:12.387116 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:14:12.387143 (+ 27us) server_negotiation.cc:407] Connection header received
0925 15:14:12.387201 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:14:12.387207 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:14:12.387412 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:14:12.387547 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:14:12.388398 (+ 851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:14:12.388405 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:14:12.388417 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:14:12.388436 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:14:12.388481 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:14:12.388496 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:14:12.388607 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 15:14:12.388685 (+ 78us) server_negotiation.cc:299] Negotiation successful
0925 15:14:12.388827 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":284,"threads_started":1}
I20260925 15:14:12.390569 22285 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:14:12.471856 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 22163
I20260925 15:14:12.488317 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 22294
I20260925 15:14:12.491677 22562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:14:12.489202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39432 (local address 127.12.8.2:46623)
0925 15:14:12.489722 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0925 15:14:12.489728 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:14:12.489755 (+ 27us) server_negotiation.cc:407] Connection header received
0925 15:14:12.489815 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:14:12.489821 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:14:12.490031 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:14:12.490158 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:14:12.491097 (+ 939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:14:12.491103 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:14:12.491114 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:14:12.491126 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:14:12.491164 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:14:12.491180 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:14:12.491294 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 15:14:12.491374 (+ 80us) server_negotiation.cc:299] Negotiation successful
0925 15:14:12.491456 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":245,"threads_started":1}
I20260925 15:14:12.492820 22416 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:14:12.572127 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 22294
I20260925 15:14:12.588671 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 22424
I20260925 15:14:12.592249 22564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:14:12.589633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41764 (local address 127.12.8.3:37513)
0925 15:14:12.590235 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0925 15:14:12.590246 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:14:12.590285 (+ 39us) server_negotiation.cc:407] Connection header received
0925 15:14:12.590354 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:14:12.590360 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:14:12.590570 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:14:12.590715 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:14:12.591686 (+ 971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:14:12.591692 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:14:12.591703 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:14:12.591714 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:14:12.591753 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:14:12.591768 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:14:12.591893 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 15:14:12.591938 (+ 45us) server_negotiation.cc:299] Negotiation successful
0925 15:14:12.592015 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":263,"threads_started":1}
I20260925 15:14:12.593408 22546 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:14:12.673159 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 22424
I20260925 15:14:12.689077 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 22073
I20260925 15:14:12.690294 22134 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:14:12.789510 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 22073
2026-09-25T15:14:12Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2322 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349162370577-12320-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:14:12 dist-test-slave-p468 krb5kdc[22569](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:14:12 dist-test-slave-p468 krb5kdc[22569](info): set up 2 sockets
Sep 25 15:14:12 dist-test-slave-p468 krb5kdc[22569](info): commencing operation
krb5kdc: starting...
W20260925 15:14:13.899668 12320 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.002s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349162370577-12320-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349162370577-12320-0/krb5kdc/test-user.keytab.
Sep 25 15:14:14 dist-test-slave-p468 krb5kdc[22569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349254, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:14:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:14:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.8.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab.
Entry for principal kudu/127.12.8.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab.
I20260925 15:14:14.129981 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349162370577-12320-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349162370577-12320-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349162370577-12320-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.62:40755
--webserver_interface=127.12.8.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab
--principal=kudu/127.12.8.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:33313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.62:40755
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:14:14.469082 22582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:14:14.469412 22582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:14:14.469571 22582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:14:14.476668 22582 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:14:14.476817 22582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:14:14.476933 22582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:14:14.477037 22582 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:14:14.477120 22582 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:14:14.486927 22582 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:33313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349162370577-12320-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349162370577-12320-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.8.62:40755
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab
--principal=kudu/127.12.8.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.62:40755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.8.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349162370577-12320-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:14:14.488277 22582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:14:14.489593 22582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:14:14.501376 22587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:14:14.501439 22588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:14:14.501628 22590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:14:14.502447 22582 server_base.cc:1034] running on GCE node
I20260925 15:14:14.503136 22582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:14:14.504444 22582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:14:14.505669 22582 hybrid_clock.cc:648] HybridClock initialized: now 1790349254505606 us; error 54 us; skew 500 ppm
I20260925 15:14:14.506047 22582 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:14:14 dist-test-slave-p468 krb5kdc[22569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349254, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:14:14.509119 22582 init.cc:377] Logged in from keytab as kudu/127.12.8.62@KRBTEST.COM (short username kudu)
I20260925 15:14:14.510505 22582 webserver.cc:460] Webserver started at http://127.12.8.62:33853/ using document root <none> and password file <none>
I20260925 15:14:14.511104 22582 fs_manager.cc:359] Metadata directory not provided
I20260925 15:14:14.511200 22582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:14:14.511448 22582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:14:14.514360 22582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349162370577-12320-0/minicluster-data/master-0/data/instance:
uuid: "8fd9c3e6e94d41379fcc1da1beac5b94"
format_stamp: "Formatted at 2026-09-25 15:14:14 on dist-test-slave-p468"
server_key: "b787d7eb44c1d85be4fd5f869ac84b5f"
server_key_iv: "d4d1a7481dc780bb7ec7aa68bdd8b3bd"
server_key_version: "encryptionkey@0"
I20260925 15:14:14.515125 22582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349162370577-12320-0/minicluster-data/master-0/wal/instance:
uuid: "8fd9c3e6e94d41379fcc1da1beac5b94"
format_stamp: "Formatted at 2026-09-25 15:14:14 on dist-test-slave-p468"
server_key: "b787d7eb44c1d85be4fd5f869ac84b5f"
server_key_iv: "d4d1a7481dc780bb7ec7aa68bdd8b3bd"
server_key_version: "encryptionkey@0"
I20260925 15:14:14.519960 22582 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 15:14:14.523185 22597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:14:14.524495 22582 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 15:14:14.524680 22582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349162370577-12320-0/minicluster-data/master-0/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349162370577-12320-0/minicluster-data/master-0/wal
uuid: "8fd9c3e6e94d41379fcc1da1beac5b94"
format_stamp: "Formatted at 2026-09-25 15:14:14 on dist-test-slave-p468"
server_key: "b787d7eb44c1d85be4fd5f869ac84b5f"
server_key_iv: "d4d1a7481dc780bb7ec7aa68bdd8b3bd"
server_key_version: "encryptionkey@0"
I20260925 15:14:14.524863 22582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349162370577-12320-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349162370577-12320-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349162370577-12320-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:14:14.546769 22582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:14:14.550899 22582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:14:14.551177 22582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:14:14.568441 22582 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.62:40755
I20260925 15:14:14.568483 22648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.62:40755 every 8 connection(s)
I20260925 15:14:14.569715 22582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
I20260925 15:14:14.572716 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 22582
I20260925 15:14:14.572937 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349162370577-12320-0/minicluster-data/master-0/wal/instance
I20260925 15:14:14.573230 22649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:14:14.573617 12320 external_mini_cluster.cc:1428] Setting key 9dadfdc16eebf271ced775acb0e26175
I20260925 15:14:14.587268 22649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fd9c3e6e94d41379fcc1da1beac5b94: Bootstrap starting.
Sep 25 15:14:14 dist-test-slave-p468 krb5kdc[22569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349254, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
I20260925 15:14:14.590286 22649 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8fd9c3e6e94d41379fcc1da1beac5b94: Neither blocks nor log segments found. Creating new log.
I20260925 15:14:14.591290 22649 log.cc:824] T 00000000000000000000000000000000 P 8fd9c3e6e94d41379fcc1da1beac5b94: Log is configured to *not* fsync() on all Append() calls
I20260925 15:14:14.593892 22649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fd9c3e6e94d41379fcc1da1beac5b94: No bootstrap required, opened a new log
I20260925 15:14:14.595486 22652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:14:14.575578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35580 (local address 127.12.8.62:40755)
0925 15:14:14.576392 (+ 814us) server_negotiation.cc:206] Beginning negotiation
0925 15:14:14.576402 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:14:14.576443 (+ 41us) server_negotiation.cc:407] Connection header received
0925 15:14:14.577933 (+ 1490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:14:14.577951 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:14:14.578404 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:14:14.578714 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:14:14.580437 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:14:14.581674 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:14:14.583438 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:14:14.584090 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:14:14.584250 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:14:14.588730 (+ 4480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:14:14.588738 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:14:14.588751 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:14:14.588787 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:14:14.591774 (+ 2987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:14:14.592475 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:14:14.592487 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:14:14.592492 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:14:14.592647 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:14:14.592998 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:14:14.593005 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:14:14.593008 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:14:14.593474 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:14:14.593770 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0925 15:14:14.594173 (+ 403us) server_negotiation.cc:299] Negotiation successful
0925 15:14:14.594603 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":401,"threads_started":1}
I20260925 15:14:14.601579 22649 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8fd9c3e6e94d41379fcc1da1beac5b94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fd9c3e6e94d41379fcc1da1beac5b94" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 40755 } }
I20260925 15:14:14.602005 22649 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8fd9c3e6e94d41379fcc1da1beac5b94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:14:14.602123 22649 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8fd9c3e6e94d41379fcc1da1beac5b94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fd9c3e6e94d41379fcc1da1beac5b94, State: Initialized, Role: FOLLOWER
I20260925 15:14:14.602713 22649 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8fd9c3e6e94d41379fcc1da1beac5b94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fd9c3e6e94d41379fcc1da1beac5b94" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 40755 } }
I20260925 15:14:14.602937 22649 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8fd9c3e6e94d41379fcc1da1beac5b94 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:14:14.603080 22649 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8fd9c3e6e94d41379fcc1da1beac5b94 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:14:14.603256 22649 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8fd9c3e6e94d41379fcc1da1beac5b94 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:14:14.605257 22649 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8fd9c3e6e94d41379fcc1da1beac5b94 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fd9c3e6e94d41379fcc1da1beac5b94" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 40755 } }
I20260925 15:14:14.605715 22649 leader_election.cc:302] T 00000000000000000000000000000000 P 8fd9c3e6e94d41379fcc1da1beac5b94 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8fd9c3e6e94d41379fcc1da1beac5b94; no voters:
I20260925 15:14:14.606312 22649 leader_election.cc:288] T 00000000000000000000000000000000 P 8fd9c3e6e94d41379fcc1da1beac5b94 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:14:14.606426 22654 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8fd9c3e6e94d41379fcc1da1beac5b94 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:14:14.607597 22654 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8fd9c3e6e94d41379fcc1da1beac5b94 [term 1 LEADER]: Becoming Leader. State: Replica: 8fd9c3e6e94d41379fcc1da1beac5b94, State: Running, Role: LEADER
I20260925 15:14:14.608116 22654 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8fd9c3e6e94d41379fcc1da1beac5b94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fd9c3e6e94d41379fcc1da1beac5b94" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 40755 } }
I20260925 15:14:14.608337 22649 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8fd9c3e6e94d41379fcc1da1beac5b94 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:14:14.610980 22656 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fd9c3e6e94d41379fcc1da1beac5b94 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8fd9c3e6e94d41379fcc1da1beac5b94. Latest consensus state: current_term: 1 leader_uuid: "8fd9c3e6e94d41379fcc1da1beac5b94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fd9c3e6e94d41379fcc1da1beac5b94" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 40755 } } }
I20260925 15:14:14.610999 22655 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fd9c3e6e94d41379fcc1da1beac5b94 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8fd9c3e6e94d41379fcc1da1beac5b94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fd9c3e6e94d41379fcc1da1beac5b94" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 40755 } } }
I20260925 15:14:14.611486 22656 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fd9c3e6e94d41379fcc1da1beac5b94 [sys.catalog]: This master's current role is: LEADER
I20260925 15:14:14.611485 22655 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fd9c3e6e94d41379fcc1da1beac5b94 [sys.catalog]: This master's current role is: LEADER
I20260925 15:14:14.613269 22662 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:14:14.618647 22662 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:14:14.627698 22662 catalog_manager.cc:1347] Generated new cluster ID: a6bbe611e5c44b9c8575fa373fd4d4e0
I20260925 15:14:14.627816 22662 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:14:14.659612 22662 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:14:14.660933 22662 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:14:14.679548 22662 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8fd9c3e6e94d41379fcc1da1beac5b94: Generated new TSK 0
I20260925 15:14:14.680401 22662 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.8.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.1.keytab.
Entry for principal kudu/127.12.8.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.1.keytab.
I20260925 15:14:14.735639 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349162370577-12320-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349162370577-12320-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349162370577-12320-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349162370577-12320-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.1:0
--local_ip_for_outbound_sockets=127.12.8.1
--webserver_interface=127.12.8.1
--webserver_port=0
--tserver_master_addrs=127.12.8.62:40755
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.1.keytab
--principal=kudu/127.12.8.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:33313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:14:15.052963 22675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:14:15.053243 22675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:14:15.053335 22675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:14:15.060293 22675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:14:15.060388 22675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:14:15.060498 22675 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.1
I20260925 15:14:15.069624 22675 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:33313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349162370577-12320-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349162370577-12320-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.1.keytab
--principal=kudu/127.12.8.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349162370577-12320-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.8.1
--webserver_port=0
--tserver_master_addrs=127.12.8.62:40755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.1
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349162370577-12320-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:14:15.070734 22675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:14:15.071928 22675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:14:15.084378 22683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:14:15.084512 22681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:14:15.084534 22680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:14:15.085415 22675 server_base.cc:1034] running on GCE node
I20260925 15:14:15.086023 22675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:14:15.086915 22675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:14:15.088119 22675 hybrid_clock.cc:648] HybridClock initialized: now 1790349255088067 us; error 35 us; skew 500 ppm
I20260925 15:14:15.088410 22675 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:14:15 dist-test-slave-p468 krb5kdc[22569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349255, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:14:15.091513 22675 init.cc:377] Logged in from keytab as kudu/127.12.8.1@KRBTEST.COM (short username kudu)
I20260925 15:14:15.092892 22675 webserver.cc:460] Webserver started at http://127.12.8.1:46595/ using document root <none> and password file <none>
I20260925 15:14:15.093540 22675 fs_manager.cc:359] Metadata directory not provided
I20260925 15:14:15.093688 22675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:14:15.094000 22675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:14:15.097456 22675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349162370577-12320-0/minicluster-data/ts-0/data/instance:
uuid: "10ec252d02de4682aa0dbf4c1647e3e0"
format_stamp: "Formatted at 2026-09-25 15:14:15 on dist-test-slave-p468"
server_key: "580973cf32eb27ccc3da61701978fd36"
server_key_iv: "0a6c7b5bdf754df62edc42b9bc71bccc"
server_key_version: "encryptionkey@0"
I20260925 15:14:15.098230 22675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349162370577-12320-0/minicluster-data/ts-0/wal/instance:
uuid: "10ec252d02de4682aa0dbf4c1647e3e0"
format_stamp: "Formatted at 2026-09-25 15:14:15 on dist-test-slave-p468"
server_key: "580973cf32eb27ccc3da61701978fd36"
server_key_iv: "0a6c7b5bdf754df62edc42b9bc71bccc"
server_key_version: "encryptionkey@0"
I20260925 15:14:15.103085 22675 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260925 15:14:15.106456 22690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:14:15.107681 22675 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 15:14:15.107848 22675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349162370577-12320-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349162370577-12320-0/minicluster-data/ts-0/wal
uuid: "10ec252d02de4682aa0dbf4c1647e3e0"
format_stamp: "Formatted at 2026-09-25 15:14:15 on dist-test-slave-p468"
server_key: "580973cf32eb27ccc3da61701978fd36"
server_key_iv: "0a6c7b5bdf754df62edc42b9bc71bccc"
server_key_version: "encryptionkey@0"
I20260925 15:14:15.108085 22675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349162370577-12320-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349162370577-12320-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349162370577-12320-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:14:15.142640 22675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:14:15.147048 22675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:14:15.147341 22675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:14:15.148279 22675 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:14:15.149578 22675 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:14:15.149681 22675 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:14:15.149827 22675 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:14:15.149900 22675 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:14:15.182403 22675 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.1:41879
I20260925 15:14:15.182436 22802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.1:41879 every 8 connection(s)
I20260925 15:14:15.183612 22675 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349162370577-12320-0/minicluster-data/ts-0/data/info.pb
I20260925 15:14:15.186141 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 22675
I20260925 15:14:15.186393 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349162370577-12320-0/minicluster-data/ts-0/wal/instance
I20260925 15:14:15.187120 12320 external_mini_cluster.cc:1428] Setting key 722359e518c10de6e9f04b5a3352d71c
Sep 25 15:14:15 dist-test-slave-p468 krb5kdc[22569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349255, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.1@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
I20260925 15:14:15.207845 22805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:14:15.186180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.1:55127 (local address 127.12.8.62:40755)
0925 15:14:15.186689 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0925 15:14:15.186696 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:14:15.187827 (+ 1131us) server_negotiation.cc:407] Connection header received
0925 15:14:15.190159 (+ 2332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:14:15.190163 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:14:15.190271 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:14:15.190361 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:14:15.192424 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:14:15.193133 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:14:15.195691 (+ 2558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:14:15.196074 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:14:15.196190 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:14:15.200238 (+ 4048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:14:15.200248 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:14:15.200257 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:14:15.200306 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:14:15.204233 (+ 3927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:14:15.205015 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:14:15.205025 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:14:15.205032 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:14:15.205190 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:14:15.205693 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:14:15.205702 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:14:15.205709 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:14:15.206141 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:14:15.206346 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0925 15:14:15.207059 (+ 713us) server_negotiation.cc:299] Negotiation successful
0925 15:14:15.207335 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":242,"threads_started":1}
I20260925 15:14:15.208436 22806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:14:15.186834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:40755 (local address 127.12.8.1:55127)
0925 15:14:15.187633 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0925 15:14:15.187690 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 15:14:15.189874 (+ 2184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:14:15.190639 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:14:15.190653 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:14:15.191451 (+ 798us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:14:15.192253 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:14:15.192279 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:14:15.193301 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:14:15.193309 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:14:15.195545 (+ 2236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:14:15.195562 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:14:15.196320 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:14:15.196329 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:14:15.196652 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:14:15.197421 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:14:15.197449 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:14:15.199999 (+ 2550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:14:15.204401 (+ 4402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:14:15.204408 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:14:15.204421 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:14:15.204862 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:14:15.205336 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:14:15.205340 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:14:15.205343 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:14:15.205542 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:14:15.206308 (+ 766us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:14:15.206314 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:14:15.206645 (+ 331us) client_negotiation.cc:769] Sending connection context
0925 15:14:15.206876 (+ 231us) client_negotiation.cc:240] Negotiation successful
0925 15:14:15.207265 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":318,"threads_started":1}
I20260925 15:14:15.210302 22803 heartbeater.cc:347] Connected to a master server at 127.12.8.62:40755
I20260925 15:14:15.210662 22803 heartbeater.cc:464] Registering TS with master...
I20260925 15:14:15.211755 22803 heartbeater.cc:511] Master 127.12.8.62:40755 requested a full tablet report, sending...
I20260925 15:14:15.214556 22614 ts_manager.cc:194] Registered new tserver with Master: 10ec252d02de4682aa0dbf4c1647e3e0 (127.12.8.1:41879)
WARNING: no policy specified for kudu/127.12.8.2@KRBTEST.COM; defaulting to no policy
I20260925 15:14:15.216490 22614 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.1@KRBTEST.COM'} at 127.12.8.1:55127
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.2.keytab.
Entry for principal kudu/127.12.8.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.2.keytab.
I20260925 15:14:15.245950 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349162370577-12320-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349162370577-12320-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349162370577-12320-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349162370577-12320-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.2:0
--local_ip_for_outbound_sockets=127.12.8.2
--webserver_interface=127.12.8.2
--webserver_port=0
--tserver_master_addrs=127.12.8.62:40755
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.2.keytab
--principal=kudu/127.12.8.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:33313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:14:15.561139 22809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:14:15.561400 22809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:14:15.561491 22809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:14:15.568396 22809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:14:15.568488 22809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:14:15.568686 22809 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.2
I20260925 15:14:15.577883 22809 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:33313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349162370577-12320-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349162370577-12320-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.2.keytab
--principal=kudu/127.12.8.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349162370577-12320-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.8.2
--webserver_port=0
--tserver_master_addrs=127.12.8.62:40755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.2
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349162370577-12320-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:14:15.579126 22809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:14:15.580410 22809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:14:15.593071 22817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:14:15.593142 22815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:14:15.593423 22814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:14:15.594391 22809 server_base.cc:1034] running on GCE node
I20260925 15:14:15.594944 22809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:14:15.595790 22809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:14:15.596976 22809 hybrid_clock.cc:648] HybridClock initialized: now 1790349255596925 us; error 53 us; skew 500 ppm
I20260925 15:14:15.597282 22809 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:14:15 dist-test-slave-p468 krb5kdc[22569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349255, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:14:15.600349 22809 init.cc:377] Logged in from keytab as kudu/127.12.8.2@KRBTEST.COM (short username kudu)
I20260925 15:14:15.601696 22809 webserver.cc:460] Webserver started at http://127.12.8.2:37613/ using document root <none> and password file <none>
I20260925 15:14:15.602378 22809 fs_manager.cc:359] Metadata directory not provided
I20260925 15:14:15.602480 22809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:14:15.602723 22809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:14:15.605436 22809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349162370577-12320-0/minicluster-data/ts-1/data/instance:
uuid: "3dd473e9a06e49078b4eaa351aefe5c0"
format_stamp: "Formatted at 2026-09-25 15:14:15 on dist-test-slave-p468"
server_key: "83afd76e0bec840c618cf2da205c04db"
server_key_iv: "1406ef0cce797724fc4363abdbe4febd"
server_key_version: "encryptionkey@0"
I20260925 15:14:15.606325 22809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349162370577-12320-0/minicluster-data/ts-1/wal/instance:
uuid: "3dd473e9a06e49078b4eaa351aefe5c0"
format_stamp: "Formatted at 2026-09-25 15:14:15 on dist-test-slave-p468"
server_key: "83afd76e0bec840c618cf2da205c04db"
server_key_iv: "1406ef0cce797724fc4363abdbe4febd"
server_key_version: "encryptionkey@0"
I20260925 15:14:15.611317 22809 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 15:14:15.614652 22824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:14:15.615901 22809 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 15:14:15.616057 22809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349162370577-12320-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349162370577-12320-0/minicluster-data/ts-1/wal
uuid: "3dd473e9a06e49078b4eaa351aefe5c0"
format_stamp: "Formatted at 2026-09-25 15:14:15 on dist-test-slave-p468"
server_key: "83afd76e0bec840c618cf2da205c04db"
server_key_iv: "1406ef0cce797724fc4363abdbe4febd"
server_key_version: "encryptionkey@0"
I20260925 15:14:15.616231 22809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349162370577-12320-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349162370577-12320-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349162370577-12320-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:14:15.650163 22809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:14:15.654078 22809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:14:15.654392 22809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:14:15.655362 22809 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:14:15.656682 22809 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:14:15.656752 22809 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:14:15.656924 22809 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:14:15.656975 22809 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:14:15.692135 22809 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.2:45217
I20260925 15:14:15.692224 22936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.2:45217 every 8 connection(s)
I20260925 15:14:15.693392 22809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349162370577-12320-0/minicluster-data/ts-1/data/info.pb
I20260925 15:14:15.696631 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 22809
I20260925 15:14:15.696858 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349162370577-12320-0/minicluster-data/ts-1/wal/instance
I20260925 15:14:15.697403 12320 external_mini_cluster.cc:1428] Setting key a985fd4421c6ae264ba6d8f00a762ef1
Sep 25 15:14:15 dist-test-slave-p468 krb5kdc[22569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349255, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.2@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
I20260925 15:14:15.713387 22805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:14:15.695811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.2:46323 (local address 127.12.8.62:40755)
0925 15:14:15.696021 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 15:14:15.696028 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:14:15.697250 (+ 1222us) server_negotiation.cc:407] Connection header received
0925 15:14:15.699412 (+ 2162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:14:15.699415 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:14:15.699508 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:14:15.699622 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:14:15.700977 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:14:15.702119 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:14:15.703645 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:14:15.704404 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:14:15.704539 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:14:15.707799 (+ 3260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:14:15.707804 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:14:15.707808 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:14:15.707838 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:14:15.710540 (+ 2702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:14:15.711243 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:14:15.711250 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:14:15.711257 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:14:15.711379 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:14:15.711785 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:14:15.711792 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:14:15.711796 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:14:15.712050 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:14:15.712202 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 15:14:15.712830 (+ 628us) server_negotiation.cc:299] Negotiation successful
0925 15:14:15.713044 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 15:14:15.714473 22939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:14:15.696311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:40755 (local address 127.12.8.2:46323)
0925 15:14:15.697081 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0925 15:14:15.697131 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 15:14:15.699146 (+ 2015us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:14:15.699803 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:14:15.699811 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:14:15.700285 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:14:15.700814 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:14:15.700830 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:14:15.702297 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:14:15.702301 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:14:15.703499 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:14:15.703510 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:14:15.704596 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:14:15.704600 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:14:15.704788 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:14:15.705456 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:14:15.705483 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:14:15.707575 (+ 2092us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:14:15.710687 (+ 3112us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:14:15.710695 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:14:15.710710 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:14:15.711126 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:14:15.711490 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:14:15.711496 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:14:15.711499 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:14:15.711676 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:14:15.712175 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:14:15.712182 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:14:15.712493 (+ 311us) client_negotiation.cc:769] Sending connection context
0925 15:14:15.712749 (+ 256us) client_negotiation.cc:240] Negotiation successful
0925 15:14:15.713280 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":329,"threads_started":1}
I20260925 15:14:15.715873 22937 heartbeater.cc:347] Connected to a master server at 127.12.8.62:40755
I20260925 15:14:15.716207 22937 heartbeater.cc:464] Registering TS with master...
I20260925 15:14:15.717127 22937 heartbeater.cc:511] Master 127.12.8.62:40755 requested a full tablet report, sending...
I20260925 15:14:15.718803 22614 ts_manager.cc:194] Registered new tserver with Master: 3dd473e9a06e49078b4eaa351aefe5c0 (127.12.8.2:45217)
I20260925 15:14:15.720007 22614 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.2@KRBTEST.COM'} at 127.12.8.2:46323
WARNING: no policy specified for kudu/127.12.8.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.3.keytab.
Entry for principal kudu/127.12.8.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.3.keytab.
I20260925 15:14:15.750406 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349162370577-12320-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349162370577-12320-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349162370577-12320-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349162370577-12320-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.3:0
--local_ip_for_outbound_sockets=127.12.8.3
--webserver_interface=127.12.8.3
--webserver_port=0
--tserver_master_addrs=127.12.8.62:40755
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.3.keytab
--principal=kudu/127.12.8.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:33313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:14:16.070852 22942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:14:16.071110 22942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:14:16.071208 22942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:14:16.078042 22942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:14:16.078140 22942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:14:16.078321 22942 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.3
I20260925 15:14:16.087498 22942 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:33313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349162370577-12320-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349162370577-12320-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.3.keytab
--principal=kudu/127.12.8.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349162370577-12320-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.8.3
--webserver_port=0
--tserver_master_addrs=127.12.8.62:40755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.3
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349162370577-12320-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:14:16.088631 22942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:14:16.089850 22942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:14:16.106662 22947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:14:16.106694 22948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:14:16.106760 22950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:14:16.107399 22942 server_base.cc:1034] running on GCE node
I20260925 15:14:16.107975 22942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:14:16.108919 22942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:14:16.110139 22942 hybrid_clock.cc:648] HybridClock initialized: now 1790349256110105 us; error 47 us; skew 500 ppm
I20260925 15:14:16.110560 22942 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:14:16 dist-test-slave-p468 krb5kdc[22569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349256, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:14:16.114003 22942 init.cc:377] Logged in from keytab as kudu/127.12.8.3@KRBTEST.COM (short username kudu)
I20260925 15:14:16.115527 22942 webserver.cc:460] Webserver started at http://127.12.8.3:36465/ using document root <none> and password file <none>
I20260925 15:14:16.116199 22942 fs_manager.cc:359] Metadata directory not provided
I20260925 15:14:16.116339 22942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:14:16.116631 22942 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:14:16.119417 22942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349162370577-12320-0/minicluster-data/ts-2/data/instance:
uuid: "5976c5f738434451a28451f551d1bfa3"
format_stamp: "Formatted at 2026-09-25 15:14:16 on dist-test-slave-p468"
server_key: "ce91ebb743f04e7f7e6f11348c697497"
server_key_iv: "7379df61f952ff5d6b94f0a85cb0646b"
server_key_version: "encryptionkey@0"
I20260925 15:14:16.120234 22942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349162370577-12320-0/minicluster-data/ts-2/wal/instance:
uuid: "5976c5f738434451a28451f551d1bfa3"
format_stamp: "Formatted at 2026-09-25 15:14:16 on dist-test-slave-p468"
server_key: "ce91ebb743f04e7f7e6f11348c697497"
server_key_iv: "7379df61f952ff5d6b94f0a85cb0646b"
server_key_version: "encryptionkey@0"
I20260925 15:14:16.125032 22942 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260925 15:14:16.128441 22957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:14:16.129992 22942 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 15:14:16.130152 22942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349162370577-12320-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349162370577-12320-0/minicluster-data/ts-2/wal
uuid: "5976c5f738434451a28451f551d1bfa3"
format_stamp: "Formatted at 2026-09-25 15:14:16 on dist-test-slave-p468"
server_key: "ce91ebb743f04e7f7e6f11348c697497"
server_key_iv: "7379df61f952ff5d6b94f0a85cb0646b"
server_key_version: "encryptionkey@0"
I20260925 15:14:16.130403 22942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349162370577-12320-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349162370577-12320-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349162370577-12320-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:14:16.149135 22942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:14:16.153252 22942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:14:16.153541 22942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:14:16.154511 22942 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:14:16.155761 22942 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:14:16.155859 22942 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:14:16.156013 22942 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:14:16.156085 22942 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:14:16.189119 22942 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.3:45247
I20260925 15:14:16.189172 23069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.3:45247 every 8 connection(s)
I20260925 15:14:16.190351 22942 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349162370577-12320-0/minicluster-data/ts-2/data/info.pb
I20260925 15:14:16.191138 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 22942
I20260925 15:14:16.191349 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349162370577-12320-0/minicluster-data/ts-2/wal/instance
I20260925 15:14:16.191927 12320 external_mini_cluster.cc:1428] Setting key e4bbc19d69da645554453b1ea6435ebd
Sep 25 15:14:16 dist-test-slave-p468 krb5kdc[22569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349256, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.3@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
I20260925 15:14:16.210445 22805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:14:16.192998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.3:48165 (local address 127.12.8.62:40755)
0925 15:14:16.193217 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 15:14:16.193224 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:14:16.194716 (+ 1492us) server_negotiation.cc:407] Connection header received
0925 15:14:16.196674 (+ 1958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:14:16.196678 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:14:16.196737 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:14:16.196817 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:14:16.198323 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:14:16.199054 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:14:16.200696 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:14:16.201069 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:14:16.201170 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:14:16.204524 (+ 3354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:14:16.204529 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:14:16.204532 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:14:16.204564 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:14:16.207704 (+ 3140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:14:16.208457 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:14:16.208461 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:14:16.208464 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:14:16.208583 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:14:16.208957 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:14:16.208960 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:14:16.208962 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:14:16.209280 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:14:16.209395 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 15:14:16.210005 (+ 610us) server_negotiation.cc:299] Negotiation successful
0925 15:14:16.210158 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 15:14:16.211452 23072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:14:16.193570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:40755 (local address 127.12.8.3:48165)
0925 15:14:16.194541 (+ 971us) negotiation.cc:107] Waiting for socket to connect
0925 15:14:16.194599 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 15:14:16.196415 (+ 1816us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:14:16.197038 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:14:16.197047 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:14:16.197568 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:14:16.198128 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:14:16.198145 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:14:16.199203 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:14:16.199207 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:14:16.200522 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:14:16.200535 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:14:16.201327 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:14:16.201332 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:14:16.201531 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:14:16.202139 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:14:16.202164 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:14:16.204340 (+ 2176us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:14:16.207815 (+ 3475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:14:16.207821 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:14:16.207832 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:14:16.208311 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:14:16.208686 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:14:16.208689 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:14:16.208691 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:14:16.208863 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:14:16.209388 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:14:16.209394 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:14:16.209684 (+ 290us) client_negotiation.cc:769] Sending connection context
0925 15:14:16.209892 (+ 208us) client_negotiation.cc:240] Negotiation successful
0925 15:14:16.210318 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"thread_start_us":336,"threads_started":1}
I20260925 15:14:16.212739 23070 heartbeater.cc:347] Connected to a master server at 127.12.8.62:40755
I20260925 15:14:16.213102 23070 heartbeater.cc:464] Registering TS with master...
I20260925 15:14:16.213815 23070 heartbeater.cc:511] Master 127.12.8.62:40755 requested a full tablet report, sending...
I20260925 15:14:16.215209 22614 ts_manager.cc:194] Registered new tserver with Master: 5976c5f738434451a28451f551d1bfa3 (127.12.8.3:45247)
I20260925 15:14:16.216003 22614 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.3@KRBTEST.COM'} at 127.12.8.3:48165
I20260925 15:14:16.219503 22803 heartbeater.cc:503] Master 127.12.8.62:40755 was elected leader, sending a full tablet report...
I20260925 15:14:16.219714 12320 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:14:16.236622 22805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:14:16.224141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35586 (local address 127.12.8.62:40755)
0925 15:14:16.224292 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 15:14:16.224297 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:14:16.224768 (+ 471us) server_negotiation.cc:407] Connection header received
0925 15:14:16.224967 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:14:16.224976 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:14:16.225032 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:14:16.225133 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:14:16.226559 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:14:16.227276 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:14:16.228585 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:14:16.229046 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:14:16.229175 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:14:16.231434 (+ 2259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:14:16.231442 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:14:16.231452 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:14:16.231482 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:14:16.233934 (+ 2452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:14:16.234686 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:14:16.234693 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:14:16.234697 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:14:16.234819 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:14:16.235209 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:14:16.235213 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:14:16.235217 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:14:16.235509 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:14:16.235781 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0925 15:14:16.236109 (+ 328us) server_negotiation.cc:299] Negotiation successful
0925 15:14:16.236314 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 15:14:16.240532 22614 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35586:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:14:16.247133 22614 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:14:16.283267 23082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:14:16.258404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.2:45217 (local address 127.0.0.1:59764)
0925 15:14:16.271771 (+ 13367us) negotiation.cc:107] Waiting for socket to connect
0925 15:14:16.271787 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:14:16.271880 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:14:16.276623 (+ 4743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:14:16.276627 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:14:16.276655 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:14:16.277094 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:14:16.277117 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:14:16.278851 (+ 1734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:14:16.278856 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:14:16.280668 (+ 1812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:14:16.280679 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:14:16.282253 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:14:16.282261 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:14:16.282445 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:14:16.282492 (+ 47us) client_negotiation.cc:769] Sending connection context
0925 15:14:16.282634 (+ 142us) client_negotiation.cc:240] Negotiation successful
0925 15:14:16.282814 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13265,"mutex_wait_us":1,"thread_start_us":13543,"threads_started":1}
I20260925 15:14:16.284026 23085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:14:16.258648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59764 (local address 127.12.8.2:45217)
0925 15:14:16.275917 (+ 17269us) server_negotiation.cc:206] Beginning negotiation
0925 15:14:16.275927 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:14:16.275959 (+ 32us) server_negotiation.cc:407] Connection header received
0925 15:14:16.276055 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:14:16.276063 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:14:16.276286 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:14:16.276465 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:14:16.277294 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:14:16.278686 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:14:16.280816 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:14:16.282046 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:14:16.282173 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:14:16.282419 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0925 15:14:16.282679 (+ 260us) server_negotiation.cc:299] Negotiation successful
0925 15:14:16.283694 (+ 1015us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17122,"thread_start_us":17032,"threads_started":1}
I20260925 15:14:16.284071 23083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:14:16.258298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.1:41879 (local address 127.0.0.1:39328)
0925 15:14:16.270857 (+ 12559us) negotiation.cc:107] Waiting for socket to connect
0925 15:14:16.270923 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0925 15:14:16.271155 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:14:16.274838 (+ 3683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:14:16.274848 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:14:16.274897 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:14:16.275362 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:14:16.275378 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:14:16.278359 (+ 2981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:14:16.278364 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:14:16.281214 (+ 2850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:14:16.281228 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:14:16.283224 (+ 1996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:14:16.283232 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:14:16.283416 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:14:16.283447 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 15:14:16.283530 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 15:14:16.283652 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12203,"mutex_wait_us":1,"thread_start_us":13232,"threads_started":1}
I20260925 15:14:16.284606 23084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:14:16.258494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39328 (local address 127.12.8.1:41879)
0925 15:14:16.273701 (+ 15207us) server_negotiation.cc:206] Beginning negotiation
0925 15:14:16.273714 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 15:14:16.273753 (+ 39us) server_negotiation.cc:407] Connection header received
0925 15:14:16.273872 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:14:16.273898 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:14:16.274251 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:14:16.274489 (+ 238us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:14:16.276616 (+ 2127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:14:16.278227 (+ 1611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:14:16.281370 (+ 3143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:14:16.282741 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:14:16.282854 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:14:16.283064 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0925 15:14:16.284189 (+ 1125us) server_negotiation.cc:299] Negotiation successful
0925 15:14:16.284374 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14964,"thread_start_us":17871,"threads_started":1}
I20260925 15:14:16.284759 23086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:14:16.272674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60418 (local address 127.12.8.3:45247)
0925 15:14:16.276338 (+ 3664us) server_negotiation.cc:206] Beginning negotiation
0925 15:14:16.276345 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:14:16.276737 (+ 392us) server_negotiation.cc:407] Connection header received
0925 15:14:16.276931 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:14:16.276941 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:14:16.277154 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:14:16.277314 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:14:16.277808 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:14:16.279283 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:14:16.281652 (+ 2369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:14:16.282863 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:14:16.282978 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:14:16.283193 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0925 15:14:16.284290 (+ 1097us) server_negotiation.cc:299] Negotiation successful
0925 15:14:16.284462 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3505,"thread_start_us":3336,"threads_started":1}
I20260925 15:14:16.285032 23087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:14:16.273153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.3:45247 (local address 127.0.0.1:60418)
0925 15:14:16.276617 (+ 3464us) negotiation.cc:107] Waiting for socket to connect
0925 15:14:16.276635 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 15:14:16.276761 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:14:16.277314 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:14:16.277318 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:14:16.277340 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:14:16.277660 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:14:16.277669 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:14:16.279445 (+ 1776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:14:16.279452 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:14:16.281505 (+ 2053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:14:16.281514 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:14:16.283454 (+ 1940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:14:16.283461 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:14:16.283615 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:14:16.283653 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 15:14:16.283739 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 15:14:16.283866 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3336,"thread_start_us":3193,"threads_started":1}
I20260925 15:14:16.287647 22872 tablet_service.cc:1467] Processing CreateTablet for tablet 2e518a722ce74431a20c3851b64450c1 (DEFAULT_TABLE table=test-table [id=c2ac644994f34ec3b755dee95c438bb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:14:16.287976 22738 tablet_service.cc:1467] Processing CreateTablet for tablet 2e518a722ce74431a20c3851b64450c1 (DEFAULT_TABLE table=test-table [id=c2ac644994f34ec3b755dee95c438bb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:14:16.288341 23005 tablet_service.cc:1467] Processing CreateTablet for tablet 2e518a722ce74431a20c3851b64450c1 (DEFAULT_TABLE table=test-table [id=c2ac644994f34ec3b755dee95c438bb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:14:16.289130 22872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e518a722ce74431a20c3851b64450c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:14:16.289326 22738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e518a722ce74431a20c3851b64450c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:14:16.289633 23005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e518a722ce74431a20c3851b64450c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:14:16.300088 23088 tablet_bootstrap.cc:492] T 2e518a722ce74431a20c3851b64450c1 P 3dd473e9a06e49078b4eaa351aefe5c0: Bootstrap starting.
I20260925 15:14:16.303391 23089 tablet_bootstrap.cc:492] T 2e518a722ce74431a20c3851b64450c1 P 5976c5f738434451a28451f551d1bfa3: Bootstrap starting.
I20260925 15:14:16.304028 23088 tablet_bootstrap.cc:654] T 2e518a722ce74431a20c3851b64450c1 P 3dd473e9a06e49078b4eaa351aefe5c0: Neither blocks nor log segments found. Creating new log.
I20260925 15:14:16.305013 23090 tablet_bootstrap.cc:492] T 2e518a722ce74431a20c3851b64450c1 P 10ec252d02de4682aa0dbf4c1647e3e0: Bootstrap starting.
I20260925 15:14:16.305505 23088 log.cc:824] T 2e518a722ce74431a20c3851b64450c1 P 3dd473e9a06e49078b4eaa351aefe5c0: Log is configured to *not* fsync() on all Append() calls
I20260925 15:14:16.307090 23089 tablet_bootstrap.cc:654] T 2e518a722ce74431a20c3851b64450c1 P 5976c5f738434451a28451f551d1bfa3: Neither blocks nor log segments found. Creating new log.
I20260925 15:14:16.308255 23089 log.cc:824] T 2e518a722ce74431a20c3851b64450c1 P 5976c5f738434451a28451f551d1bfa3: Log is configured to *not* fsync() on all Append() calls
I20260925 15:14:16.308307 23090 tablet_bootstrap.cc:654] T 2e518a722ce74431a20c3851b64450c1 P 10ec252d02de4682aa0dbf4c1647e3e0: Neither blocks nor log segments found. Creating new log.
I20260925 15:14:16.308624 23088 tablet_bootstrap.cc:492] T 2e518a722ce74431a20c3851b64450c1 P 3dd473e9a06e49078b4eaa351aefe5c0: No bootstrap required, opened a new log
I20260925 15:14:16.308948 23088 ts_tablet_manager.cc:1397] T 2e518a722ce74431a20c3851b64450c1 P 3dd473e9a06e49078b4eaa351aefe5c0: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260925 15:14:16.309366 23090 log.cc:824] T 2e518a722ce74431a20c3851b64450c1 P 10ec252d02de4682aa0dbf4c1647e3e0: Log is configured to *not* fsync() on all Append() calls
I20260925 15:14:16.315192 23090 tablet_bootstrap.cc:492] T 2e518a722ce74431a20c3851b64450c1 P 10ec252d02de4682aa0dbf4c1647e3e0: No bootstrap required, opened a new log
I20260925 15:14:16.315454 23090 ts_tablet_manager.cc:1397] T 2e518a722ce74431a20c3851b64450c1 P 10ec252d02de4682aa0dbf4c1647e3e0: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20260925 15:14:16.316499 23089 tablet_bootstrap.cc:492] T 2e518a722ce74431a20c3851b64450c1 P 5976c5f738434451a28451f551d1bfa3: No bootstrap required, opened a new log
I20260925 15:14:16.316751 23089 ts_tablet_manager.cc:1397] T 2e518a722ce74431a20c3851b64450c1 P 5976c5f738434451a28451f551d1bfa3: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.005s
I20260925 15:14:16.318360 23088 raft_consensus.cc:348] T 2e518a722ce74431a20c3851b64450c1 P 3dd473e9a06e49078b4eaa351aefe5c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ec252d02de4682aa0dbf4c1647e3e0" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 41879 } } peers { permanent_uuid: "3dd473e9a06e49078b4eaa351aefe5c0" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 45217 } } peers { permanent_uuid: "5976c5f738434451a28451f551d1bfa3" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 45247 } }
I20260925 15:14:16.318711 23088 raft_consensus.cc:374] T 2e518a722ce74431a20c3851b64450c1 P 3dd473e9a06e49078b4eaa351aefe5c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:14:16.318813 23088 raft_consensus.cc:729] T 2e518a722ce74431a20c3851b64450c1 P 3dd473e9a06e49078b4eaa351aefe5c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3dd473e9a06e49078b4eaa351aefe5c0, State: Initialized, Role: FOLLOWER
I20260925 15:14:16.319455 23088 consensus_queue.cc:260] T 2e518a722ce74431a20c3851b64450c1 P 3dd473e9a06e49078b4eaa351aefe5c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ec252d02de4682aa0dbf4c1647e3e0" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 41879 } } peers { permanent_uuid: "3dd473e9a06e49078b4eaa351aefe5c0" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 45217 } } peers { permanent_uuid: "5976c5f738434451a28451f551d1bfa3" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 45247 } }
I20260925 15:14:16.321326 23088 ts_tablet_manager.cc:1428] T 2e518a722ce74431a20c3851b64450c1 P 3dd473e9a06e49078b4eaa351aefe5c0: Time spent starting tablet: real 0.012s user 0.014s sys 0.000s
I20260925 15:14:16.321779 22937 heartbeater.cc:503] Master 127.12.8.62:40755 was elected leader, sending a full tablet report...
I20260925 15:14:16.325695 23089 raft_consensus.cc:348] T 2e518a722ce74431a20c3851b64450c1 P 5976c5f738434451a28451f551d1bfa3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ec252d02de4682aa0dbf4c1647e3e0" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 41879 } } peers { permanent_uuid: "3dd473e9a06e49078b4eaa351aefe5c0" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 45217 } } peers { permanent_uuid: "5976c5f738434451a28451f551d1bfa3" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 45247 } }
I20260925 15:14:16.326131 23089 raft_consensus.cc:374] T 2e518a722ce74431a20c3851b64450c1 P 5976c5f738434451a28451f551d1bfa3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:14:16.326267 23089 raft_consensus.cc:729] T 2e518a722ce74431a20c3851b64450c1 P 5976c5f738434451a28451f551d1bfa3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5976c5f738434451a28451f551d1bfa3, State: Initialized, Role: FOLLOWER
I20260925 15:14:16.326272 23090 raft_consensus.cc:348] T 2e518a722ce74431a20c3851b64450c1 P 10ec252d02de4682aa0dbf4c1647e3e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ec252d02de4682aa0dbf4c1647e3e0" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 41879 } } peers { permanent_uuid: "3dd473e9a06e49078b4eaa351aefe5c0" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 45217 } } peers { permanent_uuid: "5976c5f738434451a28451f551d1bfa3" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 45247 } }
I20260925 15:14:16.326727 23090 raft_consensus.cc:374] T 2e518a722ce74431a20c3851b64450c1 P 10ec252d02de4682aa0dbf4c1647e3e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:14:16.326848 23090 raft_consensus.cc:729] T 2e518a722ce74431a20c3851b64450c1 P 10ec252d02de4682aa0dbf4c1647e3e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ec252d02de4682aa0dbf4c1647e3e0, State: Initialized, Role: FOLLOWER
I20260925 15:14:16.326861 23089 consensus_queue.cc:260] T 2e518a722ce74431a20c3851b64450c1 P 5976c5f738434451a28451f551d1bfa3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ec252d02de4682aa0dbf4c1647e3e0" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 41879 } } peers { permanent_uuid: "3dd473e9a06e49078b4eaa351aefe5c0" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 45217 } } peers { permanent_uuid: "5976c5f738434451a28451f551d1bfa3" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 45247 } }
I20260925 15:14:16.327435 23090 consensus_queue.cc:260] T 2e518a722ce74431a20c3851b64450c1 P 10ec252d02de4682aa0dbf4c1647e3e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ec252d02de4682aa0dbf4c1647e3e0" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 41879 } } peers { permanent_uuid: "3dd473e9a06e49078b4eaa351aefe5c0" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 45217 } } peers { permanent_uuid: "5976c5f738434451a28451f551d1bfa3" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 45247 } }
I20260925 15:14:16.329384 23090 ts_tablet_manager.cc:1428] T 2e518a722ce74431a20c3851b64450c1 P 10ec252d02de4682aa0dbf4c1647e3e0: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260925 15:14:16.329963 23089 ts_tablet_manager.cc:1428] T 2e518a722ce74431a20c3851b64450c1 P 5976c5f738434451a28451f551d1bfa3: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260925 15:14:16.330149 23070 heartbeater.cc:503] Master 127.12.8.62:40755 was elected leader, sending a full tablet report...
W20260925 15:14:16.435988 22804 tablet.cc:2369] T 2e518a722ce74431a20c3851b64450c1 P 10ec252d02de4682aa0dbf4c1647e3e0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:14:16.442076 23071 tablet.cc:2369] T 2e518a722ce74431a20c3851b64450c1 P 5976c5f738434451a28451f551d1bfa3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:14:16.445341 22938 tablet.cc:2369] T 2e518a722ce74431a20c3851b64450c1 P 3dd473e9a06e49078b4eaa351aefe5c0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:14:16.490115 23096 raft_consensus.cc:482] T 2e518a722ce74431a20c3851b64450c1 P 10ec252d02de4682aa0dbf4c1647e3e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:14:16.490514 23096 raft_consensus.cc:504] T 2e518a722ce74431a20c3851b64450c1 P 10ec252d02de4682aa0dbf4c1647e3e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ec252d02de4682aa0dbf4c1647e3e0" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 41879 } } peers { permanent_uuid: "3dd473e9a06e49078b4eaa351aefe5c0" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 45217 } } peers { permanent_uuid: "5976c5f738434451a28451f551d1bfa3" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 45247 } }
I20260925 15:14:16.492251 23096 leader_election.cc:288] T 2e518a722ce74431a20c3851b64450c1 P 10ec252d02de4682aa0dbf4c1647e3e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3dd473e9a06e49078b4eaa351aefe5c0 (127.12.8.2:45217), 5976c5f738434451a28451f551d1bfa3 (127.12.8.3:45247)
I20260925 15:14:16.498800 23085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:14:16.492713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.1:44613 (local address 127.12.8.2:45217)
0925 15:14:16.492942 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0925 15:14:16.492951 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:14:16.493657 (+ 706us) server_negotiation.cc:407] Connection header received
0925 15:14:16.493849 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:14:16.493853 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:14:16.493919 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:14:16.494018 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:14:16.494661 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:14:16.495340 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:14:16.497052 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:14:16.497821 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:14:16.497986 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:14:16.498081 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 15:14:16.498416 (+ 335us) server_negotiation.cc:299] Negotiation successful
0925 15:14:16.498533 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 15:14:16.498983 23086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:14:16.492688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.1:59887 (local address 127.12.8.3:45247)
0925 15:14:16.492955 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0925 15:14:16.492964 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:14:16.493733 (+ 769us) server_negotiation.cc:407] Connection header received
0925 15:14:16.493947 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:14:16.493956 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:14:16.494020 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:14:16.494115 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:14:16.494827 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:14:16.495542 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:14:16.497340 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:14:16.498066 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:14:16.498234 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:14:16.498342 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 15:14:16.498611 (+ 269us) server_negotiation.cc:299] Negotiation successful
0925 15:14:16.498725 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260925 15:14:16.499141 23097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:14:16.492628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.3:45247 (local address 127.12.8.1:59887)
0925 15:14:16.493646 (+ 1018us) negotiation.cc:107] Waiting for socket to connect
0925 15:14:16.493664 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 15:14:16.493798 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:14:16.494135 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:14:16.494139 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:14:16.494175 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:14:16.494687 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:14:16.494696 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:14:16.495684 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:14:16.495689 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:14:16.497202 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:14:16.497213 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:14:16.498309 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:14:16.498314 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:14:16.498466 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:14:16.498488 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 15:14:16.498590 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 15:14:16.498714 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":892,"mutex_wait_us":79,"spinlock_wait_cycles":26880,"thread_start_us":597,"threads_started":1}
I20260925 15:14:16.499199 23098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:14:16.492628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.2:45217 (local address 127.12.8.1:44613)
0925 15:14:16.493551 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0925 15:14:16.493566 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:14:16.493692 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:14:16.494032 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:14:16.494037 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:14:16.494058 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:14:16.494519 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:14:16.494532 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:14:16.495475 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:14:16.495480 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:14:16.496905 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:14:16.496916 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:14:16.498036 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:14:16.498042 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:14:16.498215 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:14:16.498279 (+ 64us) client_negotiation.cc:769] Sending connection context
0925 15:14:16.498377 (+ 98us) client_negotiation.cc:240] Negotiation successful
0925 15:14:16.498496 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":810,"mutex_wait_us":2,"thread_start_us":555,"threads_started":1}
I20260925 15:14:16.500113 23025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e518a722ce74431a20c3851b64450c1" candidate_uuid: "10ec252d02de4682aa0dbf4c1647e3e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5976c5f738434451a28451f551d1bfa3" is_pre_election: true
I20260925 15:14:16.500120 22892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e518a722ce74431a20c3851b64450c1" candidate_uuid: "10ec252d02de4682aa0dbf4c1647e3e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3dd473e9a06e49078b4eaa351aefe5c0" is_pre_election: true
I20260925 15:14:16.500608 23025 raft_consensus.cc:2454] T 2e518a722ce74431a20c3851b64450c1 P 5976c5f738434451a28451f551d1bfa3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ec252d02de4682aa0dbf4c1647e3e0 in term 0.
I20260925 15:14:16.500669 22892 raft_consensus.cc:2454] T 2e518a722ce74431a20c3851b64450c1 P 3dd473e9a06e49078b4eaa351aefe5c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10ec252d02de4682aa0dbf4c1647e3e0 in term 0.
I20260925 15:14:16.501253 22694 leader_election.cc:302] T 2e518a722ce74431a20c3851b64450c1 P 10ec252d02de4682aa0dbf4c1647e3e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ec252d02de4682aa0dbf4c1647e3e0, 5976c5f738434451a28451f551d1bfa3; no voters:
I20260925 15:14:16.501619 23096 raft_consensus.cc:2781] T 2e518a722ce74431a20c3851b64450c1 P 10ec252d02de4682aa0dbf4c1647e3e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:14:16.501756 23096 raft_consensus.cc:482] T 2e518a722ce74431a20c3851b64450c1 P 10ec252d02de4682aa0dbf4c1647e3e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:14:16.501860 23096 raft_consensus.cc:3037] T 2e518a722ce74431a20c3851b64450c1 P 10ec252d02de4682aa0dbf4c1647e3e0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:14:16.503988 23096 raft_consensus.cc:504] T 2e518a722ce74431a20c3851b64450c1 P 10ec252d02de4682aa0dbf4c1647e3e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ec252d02de4682aa0dbf4c1647e3e0" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 41879 } } peers { permanent_uuid: "3dd473e9a06e49078b4eaa351aefe5c0" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 45217 } } peers { permanent_uuid: "5976c5f738434451a28451f551d1bfa3" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 45247 } }
I20260925 15:14:16.504592 23096 leader_election.cc:288] T 2e518a722ce74431a20c3851b64450c1 P 10ec252d02de4682aa0dbf4c1647e3e0 [CANDIDATE]: Term 1 election: Requested vote from peers 3dd473e9a06e49078b4eaa351aefe5c0 (127.12.8.2:45217), 5976c5f738434451a28451f551d1bfa3 (127.12.8.3:45247)
I20260925 15:14:16.504940 22892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e518a722ce74431a20c3851b64450c1" candidate_uuid: "10ec252d02de4682aa0dbf4c1647e3e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3dd473e9a06e49078b4eaa351aefe5c0"
I20260925 15:14:16.505045 23025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e518a722ce74431a20c3851b64450c1" candidate_uuid: "10ec252d02de4682aa0dbf4c1647e3e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5976c5f738434451a28451f551d1bfa3"
I20260925 15:14:16.505139 22892 raft_consensus.cc:3037] T 2e518a722ce74431a20c3851b64450c1 P 3dd473e9a06e49078b4eaa351aefe5c0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:14:16.505218 23025 raft_consensus.cc:3037] T 2e518a722ce74431a20c3851b64450c1 P 5976c5f738434451a28451f551d1bfa3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:14:16.508270 23025 raft_consensus.cc:2454] T 2e518a722ce74431a20c3851b64450c1 P 5976c5f738434451a28451f551d1bfa3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ec252d02de4682aa0dbf4c1647e3e0 in term 1.
I20260925 15:14:16.508270 22892 raft_consensus.cc:2454] T 2e518a722ce74431a20c3851b64450c1 P 3dd473e9a06e49078b4eaa351aefe5c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10ec252d02de4682aa0dbf4c1647e3e0 in term 1.
I20260925 15:14:16.508654 22693 leader_election.cc:302] T 2e518a722ce74431a20c3851b64450c1 P 10ec252d02de4682aa0dbf4c1647e3e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ec252d02de4682aa0dbf4c1647e3e0, 3dd473e9a06e49078b4eaa351aefe5c0; no voters:
I20260925 15:14:16.508910 23096 raft_consensus.cc:2781] T 2e518a722ce74431a20c3851b64450c1 P 10ec252d02de4682aa0dbf4c1647e3e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:14:16.509305 23096 raft_consensus.cc:686] T 2e518a722ce74431a20c3851b64450c1 P 10ec252d02de4682aa0dbf4c1647e3e0 [term 1 LEADER]: Becoming Leader. State: Replica: 10ec252d02de4682aa0dbf4c1647e3e0, State: Running, Role: LEADER
I20260925 15:14:16.509836 23096 consensus_queue.cc:237] T 2e518a722ce74431a20c3851b64450c1 P 10ec252d02de4682aa0dbf4c1647e3e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ec252d02de4682aa0dbf4c1647e3e0" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 41879 } } peers { permanent_uuid: "3dd473e9a06e49078b4eaa351aefe5c0" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 45217 } } peers { permanent_uuid: "5976c5f738434451a28451f551d1bfa3" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 45247 } }
I20260925 15:14:16.514143 22613 catalog_manager.cc:5495] T 2e518a722ce74431a20c3851b64450c1 P 10ec252d02de4682aa0dbf4c1647e3e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10ec252d02de4682aa0dbf4c1647e3e0 (127.12.8.1). New cstate: current_term: 1 leader_uuid: "10ec252d02de4682aa0dbf4c1647e3e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ec252d02de4682aa0dbf4c1647e3e0" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 41879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3dd473e9a06e49078b4eaa351aefe5c0" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 45217 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5976c5f738434451a28451f551d1bfa3" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 45247 } health_report { overall_health: UNKNOWN } } }
I20260925 15:14:16.576833 23084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:14:16.570807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39330 (local address 127.12.8.1:41879)
0925 15:14:16.570973 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 15:14:16.570978 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:14:16.571139 (+ 161us) server_negotiation.cc:407] Connection header received
0925 15:14:16.571319 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:14:16.571325 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:14:16.571424 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:14:16.571576 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:14:16.571923 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:14:16.572967 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:14:16.574382 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:14:16.575010 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:14:16.575141 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:14:16.575629 (+ 488us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:14:16.576258 (+ 629us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:14:16.576394 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 15:14:16.576494 (+ 100us) server_negotiation.cc:299] Negotiation successful
0925 15:14:16.576570 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 15:14:16.585551 22892 raft_consensus.cc:1261] T 2e518a722ce74431a20c3851b64450c1 P 3dd473e9a06e49078b4eaa351aefe5c0 [term 1 FOLLOWER]: Refusing update from remote peer 10ec252d02de4682aa0dbf4c1647e3e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:14:16.585572 23025 raft_consensus.cc:1261] T 2e518a722ce74431a20c3851b64450c1 P 5976c5f738434451a28451f551d1bfa3 [term 1 FOLLOWER]: Refusing update from remote peer 10ec252d02de4682aa0dbf4c1647e3e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:14:16.586555 23099 consensus_queue.cc:1035] T 2e518a722ce74431a20c3851b64450c1 P 10ec252d02de4682aa0dbf4c1647e3e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3dd473e9a06e49078b4eaa351aefe5c0" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 45217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:14:16.586941 23096 consensus_queue.cc:1035] T 2e518a722ce74431a20c3851b64450c1 P 10ec252d02de4682aa0dbf4c1647e3e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5976c5f738434451a28451f551d1bfa3" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:14:16.599838 23104 mvcc.cc:205] Tried to move back new op lower bound from 7333270554964680704 to 7333270554668199936. Current Snapshot: MvccSnapshot[applied={T|T < 7333270554964680704}]
I20260925 15:14:16.613018 22613 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35586:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:14:16.613386 22613 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35586:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:14:16.616781 22613 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8fd9c3e6e94d41379fcc1da1beac5b94: Sending DeleteTablet for 3 replicas of tablet 2e518a722ce74431a20c3851b64450c1
I20260925 15:14:16.617928 22872 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e518a722ce74431a20c3851b64450c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:14:16 UTC) from {username='kudu', principal='kudu/127.12.8.62@KRBTEST.COM'} at 127.0.0.1:59764
I20260925 15:14:16.617925 23005 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e518a722ce74431a20c3851b64450c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:14:16 UTC) from {username='kudu', principal='kudu/127.12.8.62@KRBTEST.COM'} at 127.0.0.1:60418
I20260925 15:14:16.618825 22738 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e518a722ce74431a20c3851b64450c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:14:16 UTC) from {username='kudu', principal='kudu/127.12.8.62@KRBTEST.COM'} at 127.0.0.1:39328
I20260925 15:14:16.619313 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 22675
I20260925 15:14:16.642302 23114 tablet_replica.cc:330] T 2e518a722ce74431a20c3851b64450c1 P 10ec252d02de4682aa0dbf4c1647e3e0: stopping tablet replica
I20260925 15:14:16.642777 23114 raft_consensus.cc:2229] T 2e518a722ce74431a20c3851b64450c1 P 10ec252d02de4682aa0dbf4c1647e3e0 [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:14:16.643427 23114 raft_consensus.cc:2258] T 2e518a722ce74431a20c3851b64450c1 P 10ec252d02de4682aa0dbf4c1647e3e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:14:16.644193 23113 tablet_replica.cc:330] T 2e518a722ce74431a20c3851b64450c1 P 5976c5f738434451a28451f551d1bfa3: stopping tablet replica
I20260925 15:14:16.644604 23113 raft_consensus.cc:2229] T 2e518a722ce74431a20c3851b64450c1 P 5976c5f738434451a28451f551d1bfa3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:14:16.644788 23112 tablet_replica.cc:330] T 2e518a722ce74431a20c3851b64450c1 P 3dd473e9a06e49078b4eaa351aefe5c0: stopping tablet replica
I20260925 15:14:16.645118 23113 raft_consensus.cc:2258] T 2e518a722ce74431a20c3851b64450c1 P 5976c5f738434451a28451f551d1bfa3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:14:16.645152 23112 raft_consensus.cc:2229] T 2e518a722ce74431a20c3851b64450c1 P 3dd473e9a06e49078b4eaa351aefe5c0 [term 1 FOLLOWER]: Raft consensus shutting down.
Sep 25 15:14:16 dist-test-slave-p468 krb5kdc[22569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349254, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.1@KRBTEST.COM
I20260925 15:14:16.646430 23112 raft_consensus.cc:2258] T 2e518a722ce74431a20c3851b64450c1 P 3dd473e9a06e49078b4eaa351aefe5c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:14:16.647377 23114 ts_tablet_manager.cc:1905] T 2e518a722ce74431a20c3851b64450c1 P 10ec252d02de4682aa0dbf4c1647e3e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:14:16.648471 23112 ts_tablet_manager.cc:1905] T 2e518a722ce74431a20c3851b64450c1 P 3dd473e9a06e49078b4eaa351aefe5c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:14:16.651234 23113 ts_tablet_manager.cc:1905] T 2e518a722ce74431a20c3851b64450c1 P 5976c5f738434451a28451f551d1bfa3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:14:16.657269 23112 ts_tablet_manager.cc:1918] T 2e518a722ce74431a20c3851b64450c1 P 3dd473e9a06e49078b4eaa351aefe5c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:14:16.657490 23112 log.cc:1196] T 2e518a722ce74431a20c3851b64450c1 P 3dd473e9a06e49078b4eaa351aefe5c0: Deleting WAL directory at /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349162370577-12320-0/minicluster-data/ts-1/wal/wals/2e518a722ce74431a20c3851b64450c1
I20260925 15:14:16.658106 23112 ts_tablet_manager.cc:1939] T 2e518a722ce74431a20c3851b64450c1 P 3dd473e9a06e49078b4eaa351aefe5c0: Deleting consensus metadata
I20260925 15:14:16.660269 23113 ts_tablet_manager.cc:1918] T 2e518a722ce74431a20c3851b64450c1 P 5976c5f738434451a28451f551d1bfa3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:14:16.660446 23113 log.cc:1196] T 2e518a722ce74431a20c3851b64450c1 P 5976c5f738434451a28451f551d1bfa3: Deleting WAL directory at /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349162370577-12320-0/minicluster-data/ts-2/wal/wals/2e518a722ce74431a20c3851b64450c1
I20260925 15:14:16.660439 22600 catalog_manager.cc:4841] TS 3dd473e9a06e49078b4eaa351aefe5c0 (127.12.8.2:45217): tablet 2e518a722ce74431a20c3851b64450c1 (table test-table [id=c2ac644994f34ec3b755dee95c438bb6]) successfully deleted
I20260925 15:14:16.660998 23113 ts_tablet_manager.cc:1939] T 2e518a722ce74431a20c3851b64450c1 P 5976c5f738434451a28451f551d1bfa3: Deleting consensus metadata
I20260925 15:14:16.664678 22601 catalog_manager.cc:4841] TS 5976c5f738434451a28451f551d1bfa3 (127.12.8.3:45247): tablet 2e518a722ce74431a20c3851b64450c1 (table test-table [id=c2ac644994f34ec3b755dee95c438bb6]) successfully deleted
I20260925 15:14:16.666574 23084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:14:16.621547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39346 (local address 127.12.8.1:41879)
0925 15:14:16.621772 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0925 15:14:16.621781 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:14:16.631834 (+ 10053us) server_negotiation.cc:407] Connection header received
0925 15:14:16.632341 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:14:16.632350 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:14:16.632495 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:14:16.632590 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:14:16.634805 (+ 2215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:14:16.636425 (+ 1620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:14:16.638944 (+ 2519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:14:16.640012 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:14:16.640138 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:14:16.646799 (+ 6661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:14:16.646818 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:14:16.646838 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:14:16.646912 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:14:16.663134 (+ 16222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:14:16.663924 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:14:16.663935 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:14:16.663939 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:14:16.664170 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:14:16.664721 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:14:16.664726 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:14:16.664728 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:14:16.665187 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:14:16.665550 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0925 15:14:16.665826 (+ 276us) server_negotiation.cc:299] Negotiation successful
0925 15:14:16.666072 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 15:14:16.667241 22798 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:14:16.777042 23114 ts_tablet_manager.cc:1918] T 2e518a722ce74431a20c3851b64450c1 P 10ec252d02de4682aa0dbf4c1647e3e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:14:16.777338 23114 log.cc:1196] T 2e518a722ce74431a20c3851b64450c1 P 10ec252d02de4682aa0dbf4c1647e3e0: Deleting WAL directory at /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349162370577-12320-0/minicluster-data/ts-0/wal/wals/2e518a722ce74431a20c3851b64450c1
I20260925 15:14:16.777944 23114 ts_tablet_manager.cc:1939] T 2e518a722ce74431a20c3851b64450c1 P 10ec252d02de4682aa0dbf4c1647e3e0: Deleting consensus metadata
I20260925 15:14:16.779592 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 22675
I20260925 15:14:16.779616 22601 catalog_manager.cc:4841] TS 10ec252d02de4682aa0dbf4c1647e3e0 (127.12.8.1:41879): tablet 2e518a722ce74431a20c3851b64450c1 (table test-table [id=c2ac644994f34ec3b755dee95c438bb6]) successfully deleted
I20260925 15:14:16.797518 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 22809
Sep 25 15:14:16 dist-test-slave-p468 krb5kdc[22569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349254, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.2@KRBTEST.COM
I20260925 15:14:16.811388 23085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:14:16.798310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59780 (local address 127.12.8.2:45217)
0925 15:14:16.798514 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 15:14:16.798520 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:14:16.798539 (+ 19us) server_negotiation.cc:407] Connection header received
0925 15:14:16.798723 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:14:16.798728 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:14:16.798797 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:14:16.798900 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:14:16.800097 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:14:16.800823 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:14:16.802019 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:14:16.802463 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:14:16.802601 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:14:16.806504 (+ 3903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:14:16.806520 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:14:16.806542 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:14:16.806575 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:14:16.809070 (+ 2495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:14:16.809762 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:14:16.809768 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:14:16.809770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:14:16.809881 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:14:16.810175 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:14:16.810178 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:14:16.810180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:14:16.810570 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:14:16.810766 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0925 15:14:16.810992 (+ 226us) server_negotiation.cc:299] Negotiation successful
0925 15:14:16.811150 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 15:14:16.812032 22932 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:14:16.899763 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 22809
I20260925 15:14:16.916853 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 22942
Sep 25 15:14:16 dist-test-slave-p468 krb5kdc[22569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349254, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.3@KRBTEST.COM
I20260925 15:14:16.932080 23086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:14:16.917548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60424 (local address 127.12.8.3:45247)
0925 15:14:16.917796 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0925 15:14:16.917803 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:14:16.917824 (+ 21us) server_negotiation.cc:407] Connection header received
0925 15:14:16.918003 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:14:16.918008 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:14:16.918075 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:14:16.918220 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:14:16.919582 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:14:16.920361 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:14:16.921713 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:14:16.922142 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:14:16.922333 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:14:16.926825 (+ 4492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:14:16.926839 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:14:16.926854 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:14:16.926884 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:14:16.929534 (+ 2650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:14:16.930181 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:14:16.930445 (+ 264us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:14:16.930447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:14:16.930588 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:14:16.930920 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:14:16.930923 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:14:16.930925 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:14:16.931223 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:14:16.931430 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0925 15:14:16.931693 (+ 263us) server_negotiation.cc:299] Negotiation successful
0925 15:14:16.931838 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260925 15:14:16.932662 23065 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:14:17.019498 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 22942
I20260925 15:14:17.035656 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 22582
I20260925 15:14:17.036736 22644 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:14:17.142134 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 22582
2026-09-25T15:14:17Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4353 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:14:17 dist-test-slave-p468 krb5kdc[23122](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:14:17 dist-test-slave-p468 krb5kdc[23122](info): set up 2 sockets
Sep 25 15:14:17 dist-test-slave-p468 krb5kdc[23122](info): commencing operation
krb5kdc: starting...
W20260925 15:14:18.262979 12320 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/krb5kdc/test-user.keytab.
Sep 25 15:14:18 dist-test-slave-p468 krb5kdc[23122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349258, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:14:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:14:18Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.12.8.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.12.8.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.12.8.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/krb5kdc/rangeradmin_127.12.8.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.12.8.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/krb5kdc/rangeradmin_127.12.8.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.12.8.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.12.8.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.12.8.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/krb5kdc/rangerlookup_127.12.8.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.12.8.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/krb5kdc/rangerlookup_127.12.8.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.12.8.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.8.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.8.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/krb5kdc/HTTP_127.12.8.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.12.8.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/krb5kdc/HTTP_127.12.8.20@KRBTEST.COM.keytab.
I20260925 15:14:18.573562 12320 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskBtce3V/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/minicluster-data/postgres -l logfile start
2026-09-25 15:14:21.081 UTC [23152] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 15:14:21.082 UTC [23152] LOG: listening on IPv4 address "127.12.8.20", port 52193
2026-09-25 15:14:21.085 UTC [23152] LOG: listening on Unix socket "/tmp/.s.PGSQL.52193"
2026-09-25 15:14:21.106 UTC [23155] LOG: database system was shut down at 2026-09-25 15:14:19 UTC
2026-09-25 15:14:21.109 UTC [23152] LOG: database system is ready to accept connections
I20260925 15:14:22.087005 12320 mini_postgres.cc:96] Postgres bound to 52193
2026-09-25 15:14:22.104 UTC [23163] FATAL: database "slave" does not exist
127.12.8.20:52193 - accepting connections
I20260925 15:14:22.104970 12320 mini_ranger.cc:161] Starting Ranger...
I20260925 15:14:22.129683 12320 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 15:14:22.634109 12320 mini_ranger.cc:87] Created ranger Postgres database
I20260925 15:14:22.634276 12320 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/minicluster-data/ranger-admin
2026-09-25 15:14:22,917 [I] DB FLAVOR :POSTGRES
2026-09-25 15:14:22,918 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 15:14:22,918 [I] Checking connection..
2026-09-25 15:14:22,918 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBtce3V/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBtce3V/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.8.20:52193/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 15:14:23,176 [I] Checking connection passed.
2026-09-25 15:14:23,176 [I] --------- Verifying version history table ---------
2026-09-25 15:14:23,176 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBtce3V/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBtce3V/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.8.20:52193/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 15:14:23,438 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 15:14:23,438 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBtce3V/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBtce3V/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.8.20:52193/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 15:14:23,681 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 15:14:23,681 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 15:14:23,682 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBtce3V/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBtce3V/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.8.20:52193/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskBtce3V/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 15:14:23.904 UTC [23227] WARNING: there is no transaction in progress
2026-09-25 15:14:23,910 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 15:14:23,911 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBtce3V/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBtce3V/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.8.20:52193/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 15:14:24,144 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 15:14:24,144 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 15:14:24,144 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBtce3V/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBtce3V/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.8.20:52193/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 15:14:24,409 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBtce3V/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBtce3V/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.8.20:52193/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 15:14:24,647 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBtce3V/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBtce3V/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.8.20:52193/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-p468.c.gcp-upstream.internal','N') ;"
2026-09-25 15:14:24,886 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBtce3V/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBtce3V/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.8.20:52193/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 15:14:25,136 [I] Table x_portal_user does not exist in database ranger
2026-09-25 15:14:25,137 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBtce3V/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBtce3V/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.8.20:52193/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 15:14:25,391 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 15:14:25,391 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBtce3V/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBtce3V/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.8.20:52193/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 15:14:25,637 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBtce3V/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBtce3V/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.8.20:52193/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 15:14:25,871 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 15:14:25,871 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBtce3V/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBtce3V/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.8.20:52193/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskBtce3V/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 15:14:26.838 UTC [23335] WARNING: there is no transaction in progress
2026-09-25 15:14:26.855 UTC [23335] WARNING: there is no transaction in progress
2026-09-25 15:14:26.872 UTC [23335] WARNING: there is no transaction in progress
2026-09-25 15:14:26.889 UTC [23335] WARNING: there is no transaction in progress
2026-09-25 15:14:26.906 UTC [23335] WARNING: there is no transaction in progress
2026-09-25 15:14:26.927 UTC [23335] WARNING: there is no transaction in progress
2026-09-25 15:14:26.980 UTC [23335] WARNING: there is no transaction in progress
2026-09-25 15:14:26.987 UTC [23335] WARNING: there is no transaction in progress
2026-09-25 15:14:27.001 UTC [23335] WARNING: there is no transaction in progress
2026-09-25 15:14:27.012 UTC [23335] WARNING: there is no transaction in progress
2026-09-25 15:14:27.022 UTC [23335] WARNING: there is no transaction in progress
2026-09-25 15:14:27.036 UTC [23335] WARNING: there is no transaction in progress
2026-09-25 15:14:27.045 UTC [23335] WARNING: there is no transaction in progress
2026-09-25 15:14:27.052 UTC [23335] WARNING: there is no transaction in progress
2026-09-25 15:14:27.059 UTC [23335] WARNING: there is no transaction in progress
2026-09-25 15:14:27.066 UTC [23335] WARNING: there is no transaction in progress
2026-09-25 15:14:27,706 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 15:14:27,706 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBtce3V/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBtce3V/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.8.20:52193/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 15:14:27,937 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 15:14:27,937 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBtce3V/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBtce3V/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.8.20:52193/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 15:14:28,192 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 15:14:28,192 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBtce3V/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBtce3V/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.8.20:52193/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 15:14:28,427 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 15:14:28,428 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBtce3V/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBtce3V/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.8.20:52193/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 15:14:28,673 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBtce3V/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBtce3V/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.8.20:52193/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 15:14:28,928 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBtce3V/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBtce3V/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.8.20:52193/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-p468.c.gcp-upstream.internal';"
2026-09-25 15:14:29,174 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 15:14:29,175 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBtce3V/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBtce3V/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.8.20:52193/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 15:14:29,418 [I] DB_PATCHES have already been applied
I20260925 15:14:29.424264 12320 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/minicluster-data/ranger-admin:/tmp/dist-test-taskBtce3V/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskBtce3V/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskBtce3V/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBtce3V/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 15:14:29.424353 12320 mini_ranger.cc:193] Using host: 127.12.8.20
I20260925 15:14:29.438500 12320 mini_ranger.cc:227] Ranger admin URL: http://127.12.8.20:51755
Sep 25, 2026 3:14:29 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 3:14:29 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 3:14:29 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 3:14:29 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 3:14:29 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 3:14:30 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 3:14:30 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 15:14:30.567951 12320 mini_ranger.cc:160] Time spent starting Ranger: real 8.463s user 0.000s sys 0.037s
Sep 25, 2026 3:14:49 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 3:14:49 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 25, 2026 3:14:49 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 3:14:49 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 3:14:49 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 3:14:49 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 3:14:49 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 3:14:49 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 3:14:49 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 3:14:49 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 3:14:49 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 3:14:49 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 3:14:49 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 3:14:49 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 3:14:49 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 3:14:49 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 3:14:49 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 3:14:50 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 15:14:51.216626 12320 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.12.8.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.12.8.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.12.8.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/krb5kdc/rangerkms_127.12.8.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.12.8.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/krb5kdc/rangerkms_127.12.8.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.12.8.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.12.8.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.8.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/krb5kdc/HTTP_127.12.8.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.12.8.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/krb5kdc/HTTP_127.12.8.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 15:14:51 dist-test-slave-p468 krb5kdc[23122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349291, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 15:14:51.408260 12320 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 15:14:51.431854 12320 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 15:14:52.288794 12320 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 15:14:52.288944 12320 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/minicluster-data/ranger-kms
I20260925 15:14:52.289170 12320 mini_ranger_kms.cc:227] Using postgres at 127.12.8.20:52193
2026-09-25 15:14:52,735 [I] DB FLAVOR :POSTGRES
2026-09-25 15:14:52,735 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 15:14:52,735 [I] Checking connection
2026-09-25 15:14:52,735 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBtce3V/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.8.20:52193/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 15:14:52,978 [I] connection success
2026-09-25 15:14:52,978 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 15:14:52,978 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 15:14:52,978 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBtce3V/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.8.20:52193/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 15:14:53,206 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 15:14:53,206 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 15:14:53,206 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 15:14:53,207 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBtce3V/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.8.20:52193/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 15:14:53,455 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 15:14:53.899920 12320 mini_ranger_kms.cc:310] Created kms service
I20260925 15:14:54.087422 12320 mini_ranger_kms.cc:326] Created kudu user
I20260925 15:14:54.131222 12320 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 15:14:54 dist-test-slave-p468 krb5kdc[23122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349291, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.12.8.20@KRBTEST.COM
I20260925 15:14:54.541878 12320 mini_ranger_kms.cc:384] Added ranger policy
I20260925 15:14:54.542245 12320 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/minicluster-data/ranger-kms:/tmp/dist-test-taskBtce3V/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBtce3V/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskBtce3V/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 15:14:54.542307 12320 mini_ranger_kms.cc:240] Using host: 127.12.8.20
I20260925 15:14:54.562263 12320 mini_ranger_kms.cc:276] Ranger KMS PID: 23515
I20260925 15:14:54.562453 12320 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.12.8.20:54245
Sep 25, 2026 3:14:55 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 3:14:55 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 3:14:55 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 3:14:55 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 3:14:55 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 3:14:55 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 25 15:14:56 dist-test-slave-p468 krb5kdc[23122](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790349296, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.8.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:14:56.870908 12320 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.463s user 0.104s sys 0.362s
I20260925 15:14:56.871157 12320 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 15:14:56.871390 12320 mini_ranger_kms.cc:401] 127.12.8.20:54245/kms/v1/keys
Sep 25 15:14:57 dist-test-slave-p468 krb5kdc[23122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349296, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.8.20@KRBTEST.COM for HTTP/127.12.8.20@KRBTEST.COM
Sep 25 15:14:57 dist-test-slave-p468 krb5kdc[23122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349296, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.8.20@KRBTEST.COM for HTTP/127.12.8.20@KRBTEST.COM
Sep 25 15:14:59 dist-test-slave-p468 krb5kdc[23122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.12.8.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab.
Entry for principal kudu/127.12.8.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab.
I20260925 15:14:59.790983 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.62:46415
--webserver_interface=127.12.8.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab
--principal=kudu/127.12.8.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:41381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.62:46415
--ranger_config_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.8.20:54245/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:15:00.118976 23566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:15:00.119246 23566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:15:00.119333 23566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:15:00.126487 23566 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:15:00.126600 23566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:15:00.126682 23566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:15:00.126785 23566 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:15:00.126869 23566 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:15:00.136341 23566 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:41381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/minicluster-data/master-0/wal
--ranger_kms_url=127.12.8.20:54245/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.12.8.62:46415
--ranger_config_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab
--principal=kudu/127.12.8.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.62:46415
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.8.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:15:00.137508 23566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:15:00.138823 23566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:15:00.150422 23574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:00.150467 23572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:00.150727 23571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:15:00.151260 23566 server_base.cc:1034] running on GCE node
I20260925 15:15:00.151957 23566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:15:00.153277 23566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:15:00.154487 23566 hybrid_clock.cc:648] HybridClock initialized: now 1790349300154449 us; error 54 us; skew 500 ppm
I20260925 15:15:00.154812 23566 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:15:00 dist-test-slave-p468 krb5kdc[23122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349300, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:15:00.165045 23566 init.cc:377] Logged in from keytab as kudu/127.12.8.62@KRBTEST.COM (short username kudu)
I20260925 15:15:00.166540 23566 webserver.cc:460] Webserver started at http://127.12.8.62:38403/ using document root <none> and password file <none>
I20260925 15:15:00.167142 23566 fs_manager.cc:359] Metadata directory not provided
I20260925 15:15:00.167232 23566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:15:00.167479 23566 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 15:15:00 dist-test-slave-p468 krb5kdc[23122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349300, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.62@KRBTEST.COM for HTTP/127.12.8.20@KRBTEST.COM
I20260925 15:15:00.222841 23566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/minicluster-data/master-0/data/instance:
uuid: "a9a934aaa14f41b5a7b5463595135c8b"
format_stamp: "Formatted at 2026-09-25 15:15:00 on dist-test-slave-p468"
server_key: "fcf79cd862cfa0f918790b58fffcd3c7"
server_key_iv: "d63f27627e81dd24f74fba374ac239e3"
server_key_version: "kuduclusterkey@0"
I20260925 15:15:00.223662 23566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/minicluster-data/master-0/wal/instance:
uuid: "a9a934aaa14f41b5a7b5463595135c8b"
format_stamp: "Formatted at 2026-09-25 15:15:00 on dist-test-slave-p468"
server_key: "fcf79cd862cfa0f918790b58fffcd3c7"
server_key_iv: "d63f27627e81dd24f74fba374ac239e3"
server_key_version: "kuduclusterkey@0"
I20260925 15:15:00.229306 23566 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260925 15:15:00.245015 23581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:00.246614 23566 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:15:00.246796 23566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/minicluster-data/master-0/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/minicluster-data/master-0/wal
uuid: "a9a934aaa14f41b5a7b5463595135c8b"
format_stamp: "Formatted at 2026-09-25 15:15:00 on dist-test-slave-p468"
server_key: "fcf79cd862cfa0f918790b58fffcd3c7"
server_key_iv: "d63f27627e81dd24f74fba374ac239e3"
server_key_version: "kuduclusterkey@0"
I20260925 15:15:00.247026 23566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:15:00.267164 23566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:15:00.270876 23566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:15:00.271185 23566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:15:00.288980 23566 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.62:46415
I20260925 15:15:00.289008 23632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.62:46415 every 8 connection(s)
I20260925 15:15:00.290338 23566 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
I20260925 15:15:00.293931 23633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:15:00.295607 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 23566
I20260925 15:15:00.295826 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/minicluster-data/master-0/wal/instance
I20260925 15:15:00.296478 12320 external_mini_cluster.cc:1428] Setting key d6ddb6f248e58ad332532172d5d6f9ed
I20260925 15:15:00.306872 23633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9a934aaa14f41b5a7b5463595135c8b: Bootstrap starting.
I20260925 15:15:00.310461 23633 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a9a934aaa14f41b5a7b5463595135c8b: Neither blocks nor log segments found. Creating new log.
I20260925 15:15:00.311497 23633 log.cc:824] T 00000000000000000000000000000000 P a9a934aaa14f41b5a7b5463595135c8b: Log is configured to *not* fsync() on all Append() calls
Sep 25 15:15:00 dist-test-slave-p468 krb5kdc[23122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
I20260925 15:15:00.314709 23633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9a934aaa14f41b5a7b5463595135c8b: No bootstrap required, opened a new log
I20260925 15:15:00.320361 23636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:00.298741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52736 (local address 127.12.8.62:46415)
0925 15:15:00.299544 (+ 803us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:00.299560 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:00.299609 (+ 49us) server_negotiation.cc:407] Connection header received
0925 15:15:00.301952 (+ 2343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:00.301980 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:00.302531 (+ 551us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:00.302968 (+ 437us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:00.304153 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:00.306110 (+ 1957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:00.307508 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:00.308551 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:00.308754 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:00.312898 (+ 4144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:00.312908 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:00.312922 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:00.312959 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:00.316102 (+ 3143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:00.316981 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:00.316988 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:00.316993 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:00.317162 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:00.317579 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:00.317582 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:00.317584 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:00.318162 (+ 578us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:00.318489 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:00.318928 (+ 439us) server_negotiation.cc:299] Negotiation successful
0925 15:15:00.319287 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":348,"threads_started":1}
I20260925 15:15:00.323295 23633 raft_consensus.cc:348] T 00000000000000000000000000000000 P a9a934aaa14f41b5a7b5463595135c8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9a934aaa14f41b5a7b5463595135c8b" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 46415 } }
I20260925 15:15:00.323660 23633 raft_consensus.cc:374] T 00000000000000000000000000000000 P a9a934aaa14f41b5a7b5463595135c8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:15:00.323767 23633 raft_consensus.cc:729] T 00000000000000000000000000000000 P a9a934aaa14f41b5a7b5463595135c8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9a934aaa14f41b5a7b5463595135c8b, State: Initialized, Role: FOLLOWER
I20260925 15:15:00.324285 23633 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9a934aaa14f41b5a7b5463595135c8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9a934aaa14f41b5a7b5463595135c8b" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 46415 } }
I20260925 15:15:00.324539 23633 raft_consensus.cc:388] T 00000000000000000000000000000000 P a9a934aaa14f41b5a7b5463595135c8b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:15:00.324700 23633 raft_consensus.cc:482] T 00000000000000000000000000000000 P a9a934aaa14f41b5a7b5463595135c8b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:15:00.324862 23633 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a9a934aaa14f41b5a7b5463595135c8b [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:15:00.327086 23633 raft_consensus.cc:504] T 00000000000000000000000000000000 P a9a934aaa14f41b5a7b5463595135c8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9a934aaa14f41b5a7b5463595135c8b" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 46415 } }
I20260925 15:15:00.327566 23633 leader_election.cc:302] T 00000000000000000000000000000000 P a9a934aaa14f41b5a7b5463595135c8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a9a934aaa14f41b5a7b5463595135c8b; no voters:
I20260925 15:15:00.328231 23633 leader_election.cc:288] T 00000000000000000000000000000000 P a9a934aaa14f41b5a7b5463595135c8b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:15:00.328374 23638 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a9a934aaa14f41b5a7b5463595135c8b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:15:00.329123 23638 raft_consensus.cc:686] T 00000000000000000000000000000000 P a9a934aaa14f41b5a7b5463595135c8b [term 1 LEADER]: Becoming Leader. State: Replica: a9a934aaa14f41b5a7b5463595135c8b, State: Running, Role: LEADER
I20260925 15:15:00.329731 23638 consensus_queue.cc:237] T 00000000000000000000000000000000 P a9a934aaa14f41b5a7b5463595135c8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9a934aaa14f41b5a7b5463595135c8b" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 46415 } }
I20260925 15:15:00.330492 23633 sys_catalog.cc:564] T 00000000000000000000000000000000 P a9a934aaa14f41b5a7b5463595135c8b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:15:00.332474 23640 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9a934aaa14f41b5a7b5463595135c8b [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9a934aaa14f41b5a7b5463595135c8b. Latest consensus state: current_term: 1 leader_uuid: "a9a934aaa14f41b5a7b5463595135c8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9a934aaa14f41b5a7b5463595135c8b" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 46415 } } }
I20260925 15:15:00.333112 23640 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9a934aaa14f41b5a7b5463595135c8b [sys.catalog]: This master's current role is: LEADER
I20260925 15:15:00.332984 23639 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9a934aaa14f41b5a7b5463595135c8b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a9a934aaa14f41b5a7b5463595135c8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9a934aaa14f41b5a7b5463595135c8b" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 46415 } } }
I20260925 15:15:00.333264 23639 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9a934aaa14f41b5a7b5463595135c8b [sys.catalog]: This master's current role is: LEADER
I20260925 15:15:00.342649 23646 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:15:00.343075 23633 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 15:15:00.354486 23646 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:15:00.365629 23646 catalog_manager.cc:1347] Generated new cluster ID: 0b51ffbc97c04b9ab4a162f58a8ec13c
I20260925 15:15:00.365765 23646 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:15:00.384997 23646 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:15:00.386984 23646 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:15:00.400482 23646 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a9a934aaa14f41b5a7b5463595135c8b: Generated new TSK 0
I20260925 15:15:00.401561 23646 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.8.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.1.keytab.
Entry for principal kudu/127.12.8.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.1.keytab.
I20260925 15:15:01.179183 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.8.20:54245/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.1:0
--local_ip_for_outbound_sockets=127.12.8.1
--webserver_interface=127.12.8.1
--webserver_port=0
--tserver_master_addrs=127.12.8.62:46415
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.1.keytab
--principal=kudu/127.12.8.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:41381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 15:15:01 dist-test-slave-p468 krb5kdc[23122](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790349301, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260925 15:15:01.596424 23678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:15:01.596822 23678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:15:01.597046 23678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:15:01.608608 23678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:15:01.608741 23678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:15:01.608982 23678 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.1
I20260925 15:15:01.624678 23678 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:41381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.12.8.20:54245/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.1.keytab
--principal=kudu/127.12.8.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.8.1
--webserver_port=0
--tserver_master_addrs=127.12.8.62:46415
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.1
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:15:01.626441 23678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:15:01.628189 23678 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:15:01.643365 23689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:01.644120 23691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:01.644502 23688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:15:01.645247 23678 server_base.cc:1034] running on GCE node
I20260925 15:15:01.645943 23678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:15:01.647037 23678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:15:01.648288 23678 hybrid_clock.cc:648] HybridClock initialized: now 1790349301648225 us; error 55 us; skew 500 ppm
I20260925 15:15:01.648633 23678 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:15:01 dist-test-slave-p468 krb5kdc[23122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349301, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:15:01.652077 23678 init.cc:377] Logged in from keytab as kudu/127.12.8.1@KRBTEST.COM (short username kudu)
I20260925 15:15:01.655480 23678 webserver.cc:460] Webserver started at http://127.12.8.1:44955/ using document root <none> and password file <none>
I20260925 15:15:01.656296 23678 fs_manager.cc:359] Metadata directory not provided
I20260925 15:15:01.656438 23678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:15:01.656761 23678 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 15:15:01 dist-test-slave-p468 krb5kdc[23122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349301, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.1@KRBTEST.COM for HTTP/127.12.8.20@KRBTEST.COM
I20260925 15:15:01.674630 23678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/minicluster-data/ts-0/data/instance:
uuid: "19a333043931427d8c586d597c30ff8b"
format_stamp: "Formatted at 2026-09-25 15:15:01 on dist-test-slave-p468"
server_key: "60b1f9ff5acffe6610ece6c0481f6260"
server_key_iv: "8ef0c403b7abcb57b53f10d5c8710fdf"
server_key_version: "kuduclusterkey@0"
I20260925 15:15:01.675621 23678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/minicluster-data/ts-0/wal/instance:
uuid: "19a333043931427d8c586d597c30ff8b"
format_stamp: "Formatted at 2026-09-25 15:15:01 on dist-test-slave-p468"
server_key: "60b1f9ff5acffe6610ece6c0481f6260"
server_key_iv: "8ef0c403b7abcb57b53f10d5c8710fdf"
server_key_version: "kuduclusterkey@0"
I20260925 15:15:01.692274 23678 fs_manager.cc:691] Time spent creating directory manager: real 0.016s user 0.006s sys 0.009s
I20260925 15:15:01.712960 23698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:01.715006 23678 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260925 15:15:01.715197 23678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/minicluster-data/ts-0/wal
uuid: "19a333043931427d8c586d597c30ff8b"
format_stamp: "Formatted at 2026-09-25 15:15:01 on dist-test-slave-p468"
server_key: "60b1f9ff5acffe6610ece6c0481f6260"
server_key_iv: "8ef0c403b7abcb57b53f10d5c8710fdf"
server_key_version: "kuduclusterkey@0"
I20260925 15:15:01.715444 23678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:15:01.741874 23678 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:15:01.747212 23678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:15:01.747548 23678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:15:01.758045 23678 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:15:01.765122 23678 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:15:01.765234 23678 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:01.765353 23678 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:15:01.765416 23678 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:01.855311 23678 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.1:44217
I20260925 15:15:01.856962 23678 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/minicluster-data/ts-0/data/info.pb
I20260925 15:15:01.855311 23810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.1:44217 every 8 connection(s)
I20260925 15:15:01.873334 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 23678
I20260925 15:15:01.873571 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/minicluster-data/ts-0/wal/instance
I20260925 15:15:01.874451 12320 external_mini_cluster.cc:1428] Setting key 4a9bd3d570e5d44c3ac6ccea6235484a
Sep 25 15:15:01 dist-test-slave-p468 krb5kdc[23122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349301, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.1@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.8.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.2@KRBTEST.COM" created.
Sep 25 15:15:01 dist-test-slave-p468 krb5kdc[23122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349301, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.62@KRBTEST.COM for HTTP/127.12.8.20@KRBTEST.COM
I20260925 15:15:01.921185 23814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:01.883861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.1:47009 (local address 127.12.8.62:46415)
0925 15:15:01.884456 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:01.884463 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:01.885545 (+ 1082us) server_negotiation.cc:407] Connection header received
0925 15:15:01.888547 (+ 3002us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:01.888553 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:01.888667 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:01.888785 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:01.896129 (+ 7344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:01.897383 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:01.905581 (+ 8198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:01.906452 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:01.906600 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:01.912743 (+ 6143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:01.912752 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:01.912760 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:01.912824 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:01.917065 (+ 4241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:01.917828 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:01.917834 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:01.917838 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:01.918005 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:01.918535 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:01.918543 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:01.918548 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:01.918946 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:01.919100 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:01.920350 (+ 1250us) server_negotiation.cc:299] Negotiation successful
0925 15:15:01.920691 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":628,"threads_started":1}
I20260925 15:15:01.922966 23813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:01.879990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:46415 (local address 127.12.8.1:47009)
0925 15:15:01.885364 (+ 5374us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:01.885409 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:01.887983 (+ 2574us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:01.889316 (+ 1333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:01.889329 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:01.890025 (+ 696us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:15:01.895906 (+ 5881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:01.895935 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:01.897602 (+ 1667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:01.897610 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:01.905358 (+ 7748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:01.905380 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:01.907250 (+ 1870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:01.907265 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:01.907754 (+ 489us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:01.908868 (+ 1114us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:15:01.908915 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:15:01.912244 (+ 3329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:15:01.917246 (+ 5002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:01.917256 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:01.917279 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:01.917691 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:01.918133 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:01.918139 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:01.918144 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:01.918410 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:01.919175 (+ 765us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:15:01.919186 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:15:01.919666 (+ 480us) client_negotiation.cc:769] Sending connection context
0925 15:15:01.919974 (+ 308us) client_negotiation.cc:240] Negotiation successful
0925 15:15:01.921240 (+ 1266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5116,"thread_start_us":11922,"threads_started":1}
I20260925 15:15:01.924950 23811 heartbeater.cc:347] Connected to a master server at 127.12.8.62:46415
I20260925 15:15:01.925264 23811 heartbeater.cc:464] Registering TS with master...
I20260925 15:15:01.926133 23811 heartbeater.cc:511] Master 127.12.8.62:46415 requested a full tablet report, sending...
I20260925 15:15:01.928879 23598 ts_manager.cc:194] Registered new tserver with Master: 19a333043931427d8c586d597c30ff8b (127.12.8.1:44217)
I20260925 15:15:01.931006 23598 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.1@KRBTEST.COM'} at 127.12.8.1:47009
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.2.keytab.
Entry for principal kudu/127.12.8.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.2.keytab.
I20260925 15:15:01.961987 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.8.20:54245/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.2:0
--local_ip_for_outbound_sockets=127.12.8.2
--webserver_interface=127.12.8.2
--webserver_port=0
--tserver_master_addrs=127.12.8.62:46415
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.2.keytab
--principal=kudu/127.12.8.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:41381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 15:15:02 dist-test-slave-p468 krb5kdc[23122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349301, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.62@KRBTEST.COM for HTTP/127.12.8.20@KRBTEST.COM
W20260925 15:15:02.454792 23817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:15:02.455274 23817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:15:02.455451 23817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:15:02.466491 23817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:15:02.466732 23817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:15:02.466977 23817 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.2
I20260925 15:15:02.482484 23817 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:41381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.12.8.20:54245/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.2.keytab
--principal=kudu/127.12.8.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.8.2
--webserver_port=0
--tserver_master_addrs=127.12.8.62:46415
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.2
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:15:02.484509 23817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:15:02.486222 23817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:15:02.521757 23825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:02.526619 23824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:02.532819 23828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:15:02.533346 23817 server_base.cc:1034] running on GCE node
I20260925 15:15:02.534374 23817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:15:02.537328 23817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:15:02.538611 23817 hybrid_clock.cc:648] HybridClock initialized: now 1790349302538514 us; error 92 us; skew 500 ppm
I20260925 15:15:02.539013 23817 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:15:02 dist-test-slave-p468 krb5kdc[23122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349302, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:15:02.549714 23817 init.cc:377] Logged in from keytab as kudu/127.12.8.2@KRBTEST.COM (short username kudu)
I20260925 15:15:02.594646 23817 webserver.cc:460] Webserver started at http://127.12.8.2:35355/ using document root <none> and password file <none>
I20260925 15:15:02.595783 23817 fs_manager.cc:359] Metadata directory not provided
I20260925 15:15:02.596014 23817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:15:02.596413 23817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 15:15:02 dist-test-slave-p468 krb5kdc[23122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349302, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.2@KRBTEST.COM for HTTP/127.12.8.20@KRBTEST.COM
I20260925 15:15:02.631964 23817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/minicluster-data/ts-1/data/instance:
uuid: "0c2446bd030f44098a26eb5bc85b6543"
format_stamp: "Formatted at 2026-09-25 15:15:02 on dist-test-slave-p468"
server_key: "96f2e6001ff58361555febbe68a3d9b2"
server_key_iv: "501c16c2d5218f07d6228023a22a0202"
server_key_version: "kuduclusterkey@0"
I20260925 15:15:02.633214 23817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/minicluster-data/ts-1/wal/instance:
uuid: "0c2446bd030f44098a26eb5bc85b6543"
format_stamp: "Formatted at 2026-09-25 15:15:02 on dist-test-slave-p468"
server_key: "96f2e6001ff58361555febbe68a3d9b2"
server_key_iv: "501c16c2d5218f07d6228023a22a0202"
server_key_version: "kuduclusterkey@0"
I20260925 15:15:02.642634 23817 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.006s sys 0.000s
I20260925 15:15:02.682078 23841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:02.684231 23817 fs_manager.cc:725] Time spent opening block manager: real 0.021s user 0.005s sys 0.000s
I20260925 15:15:02.684494 23817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/minicluster-data/ts-1/wal
uuid: "0c2446bd030f44098a26eb5bc85b6543"
format_stamp: "Formatted at 2026-09-25 15:15:02 on dist-test-slave-p468"
server_key: "96f2e6001ff58361555febbe68a3d9b2"
server_key_iv: "501c16c2d5218f07d6228023a22a0202"
server_key_version: "kuduclusterkey@0"
I20260925 15:15:02.684825 23817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:15:02.746052 23817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:15:02.751230 23817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:15:02.751551 23817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:15:02.752725 23817 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:15:02.754575 23817 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:15:02.754671 23817 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:02.754860 23817 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:15:02.754936 23817 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:02.799194 23817 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.2:46251
I20260925 15:15:02.799237 23953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.2:46251 every 8 connection(s)
I20260925 15:15:02.800786 23817 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/minicluster-data/ts-1/data/info.pb
I20260925 15:15:02.804912 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 23817
I20260925 15:15:02.805073 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/minicluster-data/ts-1/wal/instance
I20260925 15:15:02.805696 12320 external_mini_cluster.cc:1428] Setting key bcd8cc2a35dfa94b7f75c1944289f398
Sep 25 15:15:02 dist-test-slave-p468 krb5kdc[23122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349302, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.2@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.8.3@KRBTEST.COM; defaulting to no policy
I20260925 15:15:02.836696 23957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:02.806436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.2:44399 (local address 127.12.8.62:46415)
0925 15:15:02.806893 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:02.806900 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:02.815319 (+ 8419us) server_negotiation.cc:407] Connection header received
0925 15:15:02.817996 (+ 2677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:02.818003 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:02.818104 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:02.818250 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:02.820246 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:02.821362 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:02.823990 (+ 2628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:02.824595 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:02.824716 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:02.828961 (+ 4245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:02.828968 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:02.828973 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:02.829013 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:02.832932 (+ 3919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:02.833904 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:02.833912 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:02.833918 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:02.834073 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:02.834646 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:02.834654 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:02.834659 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:02.835055 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:02.835191 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:02.836102 (+ 911us) server_negotiation.cc:299] Negotiation successful
0925 15:15:02.836350 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":719,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.3@KRBTEST.COM" created.
I20260925 15:15:02.842465 23956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:02.804108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:46415 (local address 127.12.8.2:44399)
0925 15:15:02.814829 (+ 10721us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:02.814887 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:02.817612 (+ 2725us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:02.818540 (+ 928us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:02.818554 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:02.819243 (+ 689us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:15:02.820080 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:02.820106 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:02.821508 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:02.821516 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:02.823858 (+ 2342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:02.823874 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:02.824845 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:02.824854 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:02.825191 (+ 337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:02.825967 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:15:02.826000 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:15:02.828736 (+ 2736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:15:02.833104 (+ 4368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:02.833114 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:02.833132 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:02.833736 (+ 604us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:02.834259 (+ 523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:02.834264 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:02.834268 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:02.834514 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:02.835219 (+ 705us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:15:02.835227 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:15:02.835594 (+ 367us) client_negotiation.cc:769] Sending connection context
0925 15:15:02.835862 (+ 268us) client_negotiation.cc:240] Negotiation successful
0925 15:15:02.836397 (+ 535us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10364,"thread_start_us":10158,"threads_started":1}
I20260925 15:15:02.844238 23954 heartbeater.cc:347] Connected to a master server at 127.12.8.62:46415
I20260925 15:15:02.844574 23954 heartbeater.cc:464] Registering TS with master...
I20260925 15:15:02.845438 23954 heartbeater.cc:511] Master 127.12.8.62:46415 requested a full tablet report, sending...
I20260925 15:15:02.847110 23598 ts_manager.cc:194] Registered new tserver with Master: 0c2446bd030f44098a26eb5bc85b6543 (127.12.8.2:46251)
I20260925 15:15:02.848349 23598 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.2@KRBTEST.COM'} at 127.12.8.2:44399
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.3.keytab.
Entry for principal kudu/127.12.8.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.3.keytab.
I20260925 15:15:02.870007 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.8.20:54245/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.3:0
--local_ip_for_outbound_sockets=127.12.8.3
--webserver_interface=127.12.8.3
--webserver_port=0
--tserver_master_addrs=127.12.8.62:46415
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.3.keytab
--principal=kudu/127.12.8.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:41381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:15:02.935111 23811 heartbeater.cc:503] Master 127.12.8.62:46415 was elected leader, sending a full tablet report...
W20260925 15:15:03.197094 23960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:15:03.197402 23960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:15:03.197563 23960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:15:03.204800 23960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:15:03.204974 23960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:15:03.205145 23960 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.3
I20260925 15:15:03.215216 23960 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:41381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.12.8.20:54245/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.3.keytab
--principal=kudu/127.12.8.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.8.3
--webserver_port=0
--tserver_master_addrs=127.12.8.62:46415
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.3
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:15:03.216518 23960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:15:03.217772 23960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:15:03.231029 23966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:03.231002 23965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:03.231065 23968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:15:03.232129 23960 server_base.cc:1034] running on GCE node
I20260925 15:15:03.232714 23960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:15:03.233695 23960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:15:03.235003 23960 hybrid_clock.cc:648] HybridClock initialized: now 1790349303234946 us; error 62 us; skew 500 ppm
I20260925 15:15:03.235458 23960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:15:03 dist-test-slave-p468 krb5kdc[23122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349303, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:15:03.238739 23960 init.cc:377] Logged in from keytab as kudu/127.12.8.3@KRBTEST.COM (short username kudu)
I20260925 15:15:03.240211 23960 webserver.cc:460] Webserver started at http://127.12.8.3:43231/ using document root <none> and password file <none>
I20260925 15:15:03.240926 23960 fs_manager.cc:359] Metadata directory not provided
I20260925 15:15:03.241082 23960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:15:03.241412 23960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 15:15:03 dist-test-slave-p468 krb5kdc[23122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349303, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.3@KRBTEST.COM for HTTP/127.12.8.20@KRBTEST.COM
I20260925 15:15:03.256364 23960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/minicluster-data/ts-2/data/instance:
uuid: "0e3a8465c355415faa756af2b0b7f832"
format_stamp: "Formatted at 2026-09-25 15:15:03 on dist-test-slave-p468"
server_key: "6bc0693b017c5ac08cfd8d64405bcc9a"
server_key_iv: "0652c875ca93e45c307a5420aa2fce34"
server_key_version: "kuduclusterkey@0"
I20260925 15:15:03.257290 23960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/minicluster-data/ts-2/wal/instance:
uuid: "0e3a8465c355415faa756af2b0b7f832"
format_stamp: "Formatted at 2026-09-25 15:15:03 on dist-test-slave-p468"
server_key: "6bc0693b017c5ac08cfd8d64405bcc9a"
server_key_iv: "0652c875ca93e45c307a5420aa2fce34"
server_key_version: "kuduclusterkey@0"
I20260925 15:15:03.263123 23960 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260925 15:15:03.277715 23975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:03.279556 23960 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260925 15:15:03.279740 23960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/minicluster-data/ts-2/wal
uuid: "0e3a8465c355415faa756af2b0b7f832"
format_stamp: "Formatted at 2026-09-25 15:15:03 on dist-test-slave-p468"
server_key: "6bc0693b017c5ac08cfd8d64405bcc9a"
server_key_iv: "0652c875ca93e45c307a5420aa2fce34"
server_key_version: "kuduclusterkey@0"
I20260925 15:15:03.280011 23960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:15:03.297343 23960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:15:03.301419 23960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:15:03.301719 23960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:15:03.302824 23960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:15:03.304199 23960 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:15:03.304309 23960 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:03.304461 23960 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:15:03.304534 23960 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:03.339815 23960 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.3:41229
I20260925 15:15:03.339880 24087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.3:41229 every 8 connection(s)
I20260925 15:15:03.341064 23960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/minicluster-data/ts-2/data/info.pb
I20260925 15:15:03.345050 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 23960
I20260925 15:15:03.345245 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/minicluster-data/ts-2/wal/instance
I20260925 15:15:03.345817 12320 external_mini_cluster.cc:1428] Setting key 41ea43112b5670eaa6d7a74e6a71e6b0
Sep 25 15:15:03 dist-test-slave-p468 krb5kdc[23122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349303, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.3@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
I20260925 15:15:03.361617 24090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:03.343698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.3:57447 (local address 127.12.8.62:46415)
0925 15:15:03.344171 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:03.344176 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:03.346288 (+ 2112us) server_negotiation.cc:407] Connection header received
0925 15:15:03.347702 (+ 1414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:03.347709 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:03.347833 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:03.347919 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:03.349487 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:03.350302 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:03.351998 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:03.352416 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:03.352536 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:03.355929 (+ 3393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:03.355934 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:03.355937 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:03.355971 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:03.358391 (+ 2420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:03.359089 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:03.359094 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:03.359096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:03.359213 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:03.359715 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:03.359719 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:03.359721 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:03.360032 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:03.360217 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:03.361036 (+ 819us) server_negotiation.cc:299] Negotiation successful
0925 15:15:03.361255 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":243,"threads_started":1}
I20260925 15:15:03.362666 24091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:03.344254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:46415 (local address 127.12.8.3:57447)
0925 15:15:03.345149 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:03.345208 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:03.347401 (+ 2193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:03.348158 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:03.348166 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:03.348625 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:15:03.349325 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:03.349350 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:03.350441 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:03.350446 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:03.351833 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:03.351844 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:03.352620 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:03.352625 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:03.352827 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:03.353452 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:15:03.353473 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:15:03.355728 (+ 2255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:15:03.358529 (+ 2801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:03.358535 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:03.358547 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:03.358974 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:03.359331 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:03.359335 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:03.359339 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:03.359582 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:03.360179 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:15:03.360187 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:15:03.360604 (+ 417us) client_negotiation.cc:769] Sending connection context
0925 15:15:03.360955 (+ 351us) client_negotiation.cc:240] Negotiation successful
0925 15:15:03.361433 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":371,"threads_started":1}
I20260925 15:15:03.363972 24088 heartbeater.cc:347] Connected to a master server at 127.12.8.62:46415
I20260925 15:15:03.364322 24088 heartbeater.cc:464] Registering TS with master...
I20260925 15:15:03.365131 24088 heartbeater.cc:511] Master 127.12.8.62:46415 requested a full tablet report, sending...
I20260925 15:15:03.366597 23598 ts_manager.cc:194] Registered new tserver with Master: 0e3a8465c355415faa756af2b0b7f832 (127.12.8.3:41229)
I20260925 15:15:03.367347 23598 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.3@KRBTEST.COM'} at 127.12.8.3:57447
I20260925 15:15:03.373831 12320 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:15:03.391399 24090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:03.378435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33030 (local address 127.12.8.62:46415)
0925 15:15:03.378599 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:03.378603 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:03.379159 (+ 556us) server_negotiation.cc:407] Connection header received
0925 15:15:03.379382 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:03.379386 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:03.379457 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:03.379565 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:03.380944 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:03.381669 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:03.383186 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:03.383669 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:03.383809 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:03.386310 (+ 2501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:03.386315 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:03.386323 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:03.386354 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:03.388741 (+ 2387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:03.389410 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:03.389417 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:03.389421 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:03.389549 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:03.389947 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:03.389954 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:03.389957 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:03.390287 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:03.390539 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:03.390848 (+ 309us) server_negotiation.cc:299] Negotiation successful
0925 15:15:03.391082 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 15:15:03.395402 23598 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33030:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:15:03.402122 23598 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:15:03.428354 24103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:03.414486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39106 (local address 127.12.8.1:44217)
0925 15:15:03.416412 (+ 1926us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:03.416425 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:03.416460 (+ 35us) server_negotiation.cc:407] Connection header received
0925 15:15:03.416582 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:03.416593 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:03.416917 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:03.417143 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:15:03.418359 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:03.420758 (+ 2399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:03.424523 (+ 3765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:03.426741 (+ 2218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:03.426868 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:03.427102 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:03.427633 (+ 531us) server_negotiation.cc:299] Negotiation successful
0925 15:15:03.427872 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1700,"thread_start_us":1758,"threads_started":1}
I20260925 15:15:03.428395 24102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:03.414130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.1:44217 (local address 127.0.0.1:39106)
0925 15:15:03.415413 (+ 1283us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:03.415460 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:03.415669 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:03.417493 (+ 1824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:03.417504 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:03.417554 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:15:03.418054 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:03.418071 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:03.420965 (+ 2894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:03.420974 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:03.424227 (+ 3253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:03.424249 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:03.426970 (+ 2721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:03.426976 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:03.427168 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:03.427221 (+ 53us) client_negotiation.cc:769] Sending connection context
0925 15:15:03.427347 (+ 126us) client_negotiation.cc:240] Negotiation successful
0925 15:15:03.427526 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":950,"threads_started":1}
I20260925 15:15:03.432694 23746 tablet_service.cc:1467] Processing CreateTablet for tablet af19888f835243ed92a9c340c84c6510 (DEFAULT_TABLE table=test-table [id=6d006e5e11ac494daf3794704d6e4b92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:15:03.434175 23746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af19888f835243ed92a9c340c84c6510. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:15:03.434938 24102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:03.415457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.3:41229 (local address 127.0.0.1:49450)
0925 15:15:03.428655 (+ 13198us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:03.428675 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:03.428796 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:03.429405 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:03.429409 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:03.429434 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:15:03.429624 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:03.429634 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:03.431511 (+ 1877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:03.431517 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:03.432630 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:03.432645 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:03.434013 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:03.434019 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:03.434241 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:03.434278 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 15:15:03.434394 (+ 116us) client_negotiation.cc:240] Negotiation successful
0925 15:15:03.434550 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13051,"thread_start_us":10893,"threads_started":1}
I20260925 15:15:03.435003 24106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:03.415938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49450 (local address 127.12.8.3:41229)
0925 15:15:03.426309 (+ 10371us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:03.426320 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:03.428793 (+ 2473us) server_negotiation.cc:407] Connection header received
0925 15:15:03.428943 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:03.428950 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:03.429199 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:03.429432 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:15:03.429985 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:03.431373 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:03.432798 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:03.433860 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:03.433998 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:03.434255 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:03.434451 (+ 196us) server_negotiation.cc:299] Negotiation successful
0925 15:15:03.434606 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10167,"thread_start_us":10025,"threads_started":1}
I20260925 15:15:03.438562 24105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:03.414416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.2:46251 (local address 127.0.0.1:46320)
0925 15:15:03.426598 (+ 12182us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:03.426619 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:03.426748 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:03.427601 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:03.427606 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:03.427632 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:15:03.428855 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:03.428869 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:03.434343 (+ 5474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:03.434351 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:03.436289 (+ 1938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:03.436302 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:03.437787 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:03.437793 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:03.437965 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:03.437993 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 15:15:03.438100 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 15:15:03.438242 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12057,"thread_start_us":486,"threads_started":1}
I20260925 15:15:03.438630 24104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:03.415071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46320 (local address 127.12.8.2:46251)
0925 15:15:03.416148 (+ 1077us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:03.416156 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:03.426817 (+ 10661us) server_negotiation.cc:407] Connection header received
0925 15:15:03.426925 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:03.426936 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:03.427218 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:03.427408 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:15:03.429058 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:03.434226 (+ 5168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:03.436472 (+ 2246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:03.437577 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:03.437729 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:03.437908 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:03.438140 (+ 232us) server_negotiation.cc:299] Negotiation successful
0925 15:15:03.438336 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":911,"thread_start_us":1005,"threads_started":1}
I20260925 15:15:03.438795 24023 tablet_service.cc:1467] Processing CreateTablet for tablet af19888f835243ed92a9c340c84c6510 (DEFAULT_TABLE table=test-table [id=6d006e5e11ac494daf3794704d6e4b92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:15:03.440241 24023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af19888f835243ed92a9c340c84c6510. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:15:03.442016 23889 tablet_service.cc:1467] Processing CreateTablet for tablet af19888f835243ed92a9c340c84c6510 (DEFAULT_TABLE table=test-table [id=6d006e5e11ac494daf3794704d6e4b92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:15:03.443116 23889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af19888f835243ed92a9c340c84c6510. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:15:03.450798 24107 tablet_bootstrap.cc:492] T af19888f835243ed92a9c340c84c6510 P 19a333043931427d8c586d597c30ff8b: Bootstrap starting.
I20260925 15:15:03.454818 24108 tablet_bootstrap.cc:492] T af19888f835243ed92a9c340c84c6510 P 0e3a8465c355415faa756af2b0b7f832: Bootstrap starting.
I20260925 15:15:03.455111 24107 tablet_bootstrap.cc:654] T af19888f835243ed92a9c340c84c6510 P 19a333043931427d8c586d597c30ff8b: Neither blocks nor log segments found. Creating new log.
I20260925 15:15:03.456980 24107 log.cc:824] T af19888f835243ed92a9c340c84c6510 P 19a333043931427d8c586d597c30ff8b: Log is configured to *not* fsync() on all Append() calls
I20260925 15:15:03.457885 24109 tablet_bootstrap.cc:492] T af19888f835243ed92a9c340c84c6510 P 0c2446bd030f44098a26eb5bc85b6543: Bootstrap starting.
I20260925 15:15:03.459286 24108 tablet_bootstrap.cc:654] T af19888f835243ed92a9c340c84c6510 P 0e3a8465c355415faa756af2b0b7f832: Neither blocks nor log segments found. Creating new log.
I20260925 15:15:03.460736 24108 log.cc:824] T af19888f835243ed92a9c340c84c6510 P 0e3a8465c355415faa756af2b0b7f832: Log is configured to *not* fsync() on all Append() calls
I20260925 15:15:03.461786 24109 tablet_bootstrap.cc:654] T af19888f835243ed92a9c340c84c6510 P 0c2446bd030f44098a26eb5bc85b6543: Neither blocks nor log segments found. Creating new log.
I20260925 15:15:03.462915 24109 log.cc:824] T af19888f835243ed92a9c340c84c6510 P 0c2446bd030f44098a26eb5bc85b6543: Log is configured to *not* fsync() on all Append() calls
I20260925 15:15:03.464054 24107 tablet_bootstrap.cc:492] T af19888f835243ed92a9c340c84c6510 P 19a333043931427d8c586d597c30ff8b: No bootstrap required, opened a new log
I20260925 15:15:03.464051 24108 tablet_bootstrap.cc:492] T af19888f835243ed92a9c340c84c6510 P 0e3a8465c355415faa756af2b0b7f832: No bootstrap required, opened a new log
I20260925 15:15:03.464344 24108 ts_tablet_manager.cc:1397] T af19888f835243ed92a9c340c84c6510 P 0e3a8465c355415faa756af2b0b7f832: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.005s
I20260925 15:15:03.464344 24107 ts_tablet_manager.cc:1397] T af19888f835243ed92a9c340c84c6510 P 19a333043931427d8c586d597c30ff8b: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.008s
I20260925 15:15:03.465421 24109 tablet_bootstrap.cc:492] T af19888f835243ed92a9c340c84c6510 P 0c2446bd030f44098a26eb5bc85b6543: No bootstrap required, opened a new log
I20260925 15:15:03.465677 24109 ts_tablet_manager.cc:1397] T af19888f835243ed92a9c340c84c6510 P 0c2446bd030f44098a26eb5bc85b6543: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260925 15:15:03.472834 24109 raft_consensus.cc:348] T af19888f835243ed92a9c340c84c6510 P 0c2446bd030f44098a26eb5bc85b6543 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19a333043931427d8c586d597c30ff8b" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 44217 } } peers { permanent_uuid: "0c2446bd030f44098a26eb5bc85b6543" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 46251 } } peers { permanent_uuid: "0e3a8465c355415faa756af2b0b7f832" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 41229 } }
I20260925 15:15:03.473219 24109 raft_consensus.cc:374] T af19888f835243ed92a9c340c84c6510 P 0c2446bd030f44098a26eb5bc85b6543 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:15:03.473359 24109 raft_consensus.cc:729] T af19888f835243ed92a9c340c84c6510 P 0c2446bd030f44098a26eb5bc85b6543 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c2446bd030f44098a26eb5bc85b6543, State: Initialized, Role: FOLLOWER
I20260925 15:15:03.473995 24109 consensus_queue.cc:260] T af19888f835243ed92a9c340c84c6510 P 0c2446bd030f44098a26eb5bc85b6543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19a333043931427d8c586d597c30ff8b" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 44217 } } peers { permanent_uuid: "0c2446bd030f44098a26eb5bc85b6543" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 46251 } } peers { permanent_uuid: "0e3a8465c355415faa756af2b0b7f832" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 41229 } }
I20260925 15:15:03.475128 23954 heartbeater.cc:503] Master 127.12.8.62:46415 was elected leader, sending a full tablet report...
I20260925 15:15:03.475123 24107 raft_consensus.cc:348] T af19888f835243ed92a9c340c84c6510 P 19a333043931427d8c586d597c30ff8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19a333043931427d8c586d597c30ff8b" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 44217 } } peers { permanent_uuid: "0c2446bd030f44098a26eb5bc85b6543" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 46251 } } peers { permanent_uuid: "0e3a8465c355415faa756af2b0b7f832" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 41229 } }
I20260925 15:15:03.475175 24108 raft_consensus.cc:348] T af19888f835243ed92a9c340c84c6510 P 0e3a8465c355415faa756af2b0b7f832 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19a333043931427d8c586d597c30ff8b" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 44217 } } peers { permanent_uuid: "0c2446bd030f44098a26eb5bc85b6543" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 46251 } } peers { permanent_uuid: "0e3a8465c355415faa756af2b0b7f832" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 41229 } }
I20260925 15:15:03.475558 24107 raft_consensus.cc:374] T af19888f835243ed92a9c340c84c6510 P 19a333043931427d8c586d597c30ff8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:15:03.475560 24108 raft_consensus.cc:374] T af19888f835243ed92a9c340c84c6510 P 0e3a8465c355415faa756af2b0b7f832 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:15:03.475651 24108 raft_consensus.cc:729] T af19888f835243ed92a9c340c84c6510 P 0e3a8465c355415faa756af2b0b7f832 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e3a8465c355415faa756af2b0b7f832, State: Initialized, Role: FOLLOWER
I20260925 15:15:03.475651 24107 raft_consensus.cc:729] T af19888f835243ed92a9c340c84c6510 P 19a333043931427d8c586d597c30ff8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19a333043931427d8c586d597c30ff8b, State: Initialized, Role: FOLLOWER
I20260925 15:15:03.476094 24109 ts_tablet_manager.cc:1428] T af19888f835243ed92a9c340c84c6510 P 0c2446bd030f44098a26eb5bc85b6543: Time spent starting tablet: real 0.010s user 0.008s sys 0.004s
I20260925 15:15:03.476253 24107 consensus_queue.cc:260] T af19888f835243ed92a9c340c84c6510 P 19a333043931427d8c586d597c30ff8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19a333043931427d8c586d597c30ff8b" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 44217 } } peers { permanent_uuid: "0c2446bd030f44098a26eb5bc85b6543" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 46251 } } peers { permanent_uuid: "0e3a8465c355415faa756af2b0b7f832" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 41229 } }
I20260925 15:15:03.476342 24108 consensus_queue.cc:260] T af19888f835243ed92a9c340c84c6510 P 0e3a8465c355415faa756af2b0b7f832 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19a333043931427d8c586d597c30ff8b" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 44217 } } peers { permanent_uuid: "0c2446bd030f44098a26eb5bc85b6543" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 46251 } } peers { permanent_uuid: "0e3a8465c355415faa756af2b0b7f832" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 41229 } }
I20260925 15:15:03.478111 24107 ts_tablet_manager.cc:1428] T af19888f835243ed92a9c340c84c6510 P 19a333043931427d8c586d597c30ff8b: Time spent starting tablet: real 0.014s user 0.008s sys 0.004s
I20260925 15:15:03.478659 24088 heartbeater.cc:503] Master 127.12.8.62:46415 was elected leader, sending a full tablet report...
I20260925 15:15:03.480809 24108 ts_tablet_manager.cc:1428] T af19888f835243ed92a9c340c84c6510 P 0e3a8465c355415faa756af2b0b7f832: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
W20260925 15:15:03.560220 23955 tablet.cc:2369] T af19888f835243ed92a9c340c84c6510 P 0c2446bd030f44098a26eb5bc85b6543: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:15:03.582628 24113 raft_consensus.cc:482] T af19888f835243ed92a9c340c84c6510 P 0c2446bd030f44098a26eb5bc85b6543 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:15:03.582944 24113 raft_consensus.cc:504] T af19888f835243ed92a9c340c84c6510 P 0c2446bd030f44098a26eb5bc85b6543 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19a333043931427d8c586d597c30ff8b" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 44217 } } peers { permanent_uuid: "0c2446bd030f44098a26eb5bc85b6543" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 46251 } } peers { permanent_uuid: "0e3a8465c355415faa756af2b0b7f832" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 41229 } }
I20260925 15:15:03.584820 24113 leader_election.cc:288] T af19888f835243ed92a9c340c84c6510 P 0c2446bd030f44098a26eb5bc85b6543 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 19a333043931427d8c586d597c30ff8b (127.12.8.1:44217), 0e3a8465c355415faa756af2b0b7f832 (127.12.8.3:41229)
I20260925 15:15:03.592007 24106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:03.585133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.2:59271 (local address 127.12.8.3:41229)
0925 15:15:03.585390 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:03.585397 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:03.586396 (+ 999us) server_negotiation.cc:407] Connection header received
0925 15:15:03.586618 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:03.586625 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:03.586700 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:03.587047 (+ 347us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:15:03.587678 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:03.588401 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:03.590220 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:03.591011 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:03.591177 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:03.591284 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:03.591632 (+ 348us) server_negotiation.cc:299] Negotiation successful
0925 15:15:03.591750 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 15:15:03.592082 24103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:03.585404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.2:32785 (local address 127.12.8.1:44217)
0925 15:15:03.585638 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:03.585644 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:03.586516 (+ 872us) server_negotiation.cc:407] Connection header received
0925 15:15:03.586710 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:03.586722 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:03.586799 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:03.586895 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:15:03.587546 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:03.588268 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:03.590227 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:03.591130 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:03.591314 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:03.591423 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:03.591704 (+ 281us) server_negotiation.cc:299] Negotiation successful
0925 15:15:03.591837 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 15:15:03.592175 24116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:03.585064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.3:41229 (local address 127.12.8.2:59271)
0925 15:15:03.586263 (+ 1199us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:03.586294 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:03.586478 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:03.587073 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:03.587077 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:03.587116 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:15:03.587535 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:03.587544 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:03.588546 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:03.588554 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:03.590053 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:03.590069 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:03.591226 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:03.591232 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:03.591462 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:03.591486 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 15:15:03.591607 (+ 121us) client_negotiation.cc:240] Negotiation successful
0925 15:15:03.591731 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":994,"mutex_wait_us":23,"thread_start_us":525,"threads_started":1}
I20260925 15:15:03.592217 24117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:03.585267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.1:44217 (local address 127.12.8.2:32785)
0925 15:15:03.586380 (+ 1113us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:03.586422 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:03.586567 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:03.586922 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:03.586926 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:03.586950 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:15:03.587383 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:03.587391 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:03.588395 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:03.588403 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:03.590040 (+ 1637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:03.590051 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:03.591389 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:03.591396 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:03.591546 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:03.591568 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 15:15:03.591668 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 15:15:03.591785 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":995,"mutex_wait_us":42,"spinlock_wait_cycles":17664,"thread_start_us":763,"threads_started":1}
W20260925 15:15:03.592708 24089 tablet.cc:2369] T af19888f835243ed92a9c340c84c6510 P 0e3a8465c355415faa756af2b0b7f832: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:15:03.593465 23766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af19888f835243ed92a9c340c84c6510" candidate_uuid: "0c2446bd030f44098a26eb5bc85b6543" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19a333043931427d8c586d597c30ff8b" is_pre_election: true
I20260925 15:15:03.593811 24043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af19888f835243ed92a9c340c84c6510" candidate_uuid: "0c2446bd030f44098a26eb5bc85b6543" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e3a8465c355415faa756af2b0b7f832" is_pre_election: true
I20260925 15:15:03.594081 23766 raft_consensus.cc:2454] T af19888f835243ed92a9c340c84c6510 P 19a333043931427d8c586d597c30ff8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c2446bd030f44098a26eb5bc85b6543 in term 0.
I20260925 15:15:03.594329 24043 raft_consensus.cc:2454] T af19888f835243ed92a9c340c84c6510 P 0e3a8465c355415faa756af2b0b7f832 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c2446bd030f44098a26eb5bc85b6543 in term 0.
I20260925 15:15:03.594899 23843 leader_election.cc:302] T af19888f835243ed92a9c340c84c6510 P 0c2446bd030f44098a26eb5bc85b6543 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c2446bd030f44098a26eb5bc85b6543, 19a333043931427d8c586d597c30ff8b; no voters:
I20260925 15:15:03.595299 24113 raft_consensus.cc:2781] T af19888f835243ed92a9c340c84c6510 P 0c2446bd030f44098a26eb5bc85b6543 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:15:03.595455 24113 raft_consensus.cc:482] T af19888f835243ed92a9c340c84c6510 P 0c2446bd030f44098a26eb5bc85b6543 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:15:03.595563 24113 raft_consensus.cc:3037] T af19888f835243ed92a9c340c84c6510 P 0c2446bd030f44098a26eb5bc85b6543 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:15:03.597687 24113 raft_consensus.cc:504] T af19888f835243ed92a9c340c84c6510 P 0c2446bd030f44098a26eb5bc85b6543 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19a333043931427d8c586d597c30ff8b" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 44217 } } peers { permanent_uuid: "0c2446bd030f44098a26eb5bc85b6543" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 46251 } } peers { permanent_uuid: "0e3a8465c355415faa756af2b0b7f832" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 41229 } }
I20260925 15:15:03.598312 24113 leader_election.cc:288] T af19888f835243ed92a9c340c84c6510 P 0c2446bd030f44098a26eb5bc85b6543 [CANDIDATE]: Term 1 election: Requested vote from peers 19a333043931427d8c586d597c30ff8b (127.12.8.1:44217), 0e3a8465c355415faa756af2b0b7f832 (127.12.8.3:41229)
I20260925 15:15:03.598726 23766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af19888f835243ed92a9c340c84c6510" candidate_uuid: "0c2446bd030f44098a26eb5bc85b6543" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19a333043931427d8c586d597c30ff8b"
I20260925 15:15:03.598806 24043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af19888f835243ed92a9c340c84c6510" candidate_uuid: "0c2446bd030f44098a26eb5bc85b6543" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e3a8465c355415faa756af2b0b7f832"
I20260925 15:15:03.598953 23766 raft_consensus.cc:3037] T af19888f835243ed92a9c340c84c6510 P 19a333043931427d8c586d597c30ff8b [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:15:03.598980 24043 raft_consensus.cc:3037] T af19888f835243ed92a9c340c84c6510 P 0e3a8465c355415faa756af2b0b7f832 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:15:03.602074 24043 raft_consensus.cc:2454] T af19888f835243ed92a9c340c84c6510 P 0e3a8465c355415faa756af2b0b7f832 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c2446bd030f44098a26eb5bc85b6543 in term 1.
I20260925 15:15:03.602074 23766 raft_consensus.cc:2454] T af19888f835243ed92a9c340c84c6510 P 19a333043931427d8c586d597c30ff8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c2446bd030f44098a26eb5bc85b6543 in term 1.
I20260925 15:15:03.602514 23842 leader_election.cc:302] T af19888f835243ed92a9c340c84c6510 P 0c2446bd030f44098a26eb5bc85b6543 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c2446bd030f44098a26eb5bc85b6543, 0e3a8465c355415faa756af2b0b7f832; no voters:
I20260925 15:15:03.602845 24113 raft_consensus.cc:2781] T af19888f835243ed92a9c340c84c6510 P 0c2446bd030f44098a26eb5bc85b6543 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:15:03.603287 24113 raft_consensus.cc:686] T af19888f835243ed92a9c340c84c6510 P 0c2446bd030f44098a26eb5bc85b6543 [term 1 LEADER]: Becoming Leader. State: Replica: 0c2446bd030f44098a26eb5bc85b6543, State: Running, Role: LEADER
I20260925 15:15:03.603762 24113 consensus_queue.cc:237] T af19888f835243ed92a9c340c84c6510 P 0c2446bd030f44098a26eb5bc85b6543 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19a333043931427d8c586d597c30ff8b" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 44217 } } peers { permanent_uuid: "0c2446bd030f44098a26eb5bc85b6543" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 46251 } } peers { permanent_uuid: "0e3a8465c355415faa756af2b0b7f832" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 41229 } }
I20260925 15:15:03.608225 23596 catalog_manager.cc:5495] T af19888f835243ed92a9c340c84c6510 P 0c2446bd030f44098a26eb5bc85b6543 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c2446bd030f44098a26eb5bc85b6543 (127.12.8.2). New cstate: current_term: 1 leader_uuid: "0c2446bd030f44098a26eb5bc85b6543" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19a333043931427d8c586d597c30ff8b" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 44217 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c2446bd030f44098a26eb5bc85b6543" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 46251 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e3a8465c355415faa756af2b0b7f832" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 41229 } health_report { overall_health: UNKNOWN } } }
I20260925 15:15:03.632822 24104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:03.627322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46334 (local address 127.12.8.2:46251)
0925 15:15:03.627509 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:03.627514 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:03.627567 (+ 53us) server_negotiation.cc:407] Connection header received
0925 15:15:03.627767 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:03.627770 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:03.627857 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:03.627992 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:15:03.628352 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:03.629064 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:03.630646 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:03.631178 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:03.631270 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:03.631752 (+ 482us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:15:03.632293 (+ 541us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:15:03.632415 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:03.632534 (+ 119us) server_negotiation.cc:299] Negotiation successful
0925 15:15:03.632596 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260925 15:15:03.636379 23812 tablet.cc:2369] T af19888f835243ed92a9c340c84c6510 P 19a333043931427d8c586d597c30ff8b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:15:03.640833 23766 raft_consensus.cc:1261] T af19888f835243ed92a9c340c84c6510 P 19a333043931427d8c586d597c30ff8b [term 1 FOLLOWER]: Refusing update from remote peer 0c2446bd030f44098a26eb5bc85b6543: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:15:03.640852 24043 raft_consensus.cc:1261] T af19888f835243ed92a9c340c84c6510 P 0e3a8465c355415faa756af2b0b7f832 [term 1 FOLLOWER]: Refusing update from remote peer 0c2446bd030f44098a26eb5bc85b6543: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:15:03.641760 24118 consensus_queue.cc:1035] T af19888f835243ed92a9c340c84c6510 P 0c2446bd030f44098a26eb5bc85b6543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e3a8465c355415faa756af2b0b7f832" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 41229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:15:03.642112 24113 consensus_queue.cc:1035] T af19888f835243ed92a9c340c84c6510 P 0c2446bd030f44098a26eb5bc85b6543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19a333043931427d8c586d597c30ff8b" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 44217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:15:03.664368 23596 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33030:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:15:03.664654 23596 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33030:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:15:03.667951 23596 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a9a934aaa14f41b5a7b5463595135c8b: Sending DeleteTablet for 3 replicas of tablet af19888f835243ed92a9c340c84c6510
I20260925 15:15:03.669044 23889 tablet_service.cc:1514] Processing DeleteTablet for tablet af19888f835243ed92a9c340c84c6510 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:15:03 UTC) from {username='kudu', principal='kudu/127.12.8.62@KRBTEST.COM'} at 127.0.0.1:46320
I20260925 15:15:03.669029 23746 tablet_service.cc:1514] Processing DeleteTablet for tablet af19888f835243ed92a9c340c84c6510 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:15:03 UTC) from {username='kudu', principal='kudu/127.12.8.62@KRBTEST.COM'} at 127.0.0.1:39106
I20260925 15:15:03.669914 24130 tablet_replica.cc:330] T af19888f835243ed92a9c340c84c6510 P 19a333043931427d8c586d597c30ff8b: stopping tablet replica
I20260925 15:15:03.669920 24023 tablet_service.cc:1514] Processing DeleteTablet for tablet af19888f835243ed92a9c340c84c6510 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:15:03 UTC) from {username='kudu', principal='kudu/127.12.8.62@KRBTEST.COM'} at 127.0.0.1:49450
I20260925 15:15:03.670437 24130 raft_consensus.cc:2229] T af19888f835243ed92a9c340c84c6510 P 19a333043931427d8c586d597c30ff8b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:15:03.671034 24130 raft_consensus.cc:2258] T af19888f835243ed92a9c340c84c6510 P 19a333043931427d8c586d597c30ff8b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:15:03.671244 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 23678
I20260925 15:15:03.673352 24130 ts_tablet_manager.cc:1905] T af19888f835243ed92a9c340c84c6510 P 19a333043931427d8c586d597c30ff8b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:15:03.680831 24132 tablet_replica.cc:330] T af19888f835243ed92a9c340c84c6510 P 0e3a8465c355415faa756af2b0b7f832: stopping tablet replica
I20260925 15:15:03.681334 24132 raft_consensus.cc:2229] T af19888f835243ed92a9c340c84c6510 P 0e3a8465c355415faa756af2b0b7f832 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:15:03.681854 24132 raft_consensus.cc:2258] T af19888f835243ed92a9c340c84c6510 P 0e3a8465c355415faa756af2b0b7f832 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:15:03.681975 24130 ts_tablet_manager.cc:1918] T af19888f835243ed92a9c340c84c6510 P 19a333043931427d8c586d597c30ff8b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:15:03.682156 24130 log.cc:1196] T af19888f835243ed92a9c340c84c6510 P 19a333043931427d8c586d597c30ff8b: Deleting WAL directory at /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/minicluster-data/ts-0/wal/wals/af19888f835243ed92a9c340c84c6510
I20260925 15:15:03.682900 24130 ts_tablet_manager.cc:1939] T af19888f835243ed92a9c340c84c6510 P 19a333043931427d8c586d597c30ff8b: Deleting consensus metadata
I20260925 15:15:03.683941 24132 ts_tablet_manager.cc:1905] T af19888f835243ed92a9c340c84c6510 P 0e3a8465c355415faa756af2b0b7f832: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:15:03.684762 23583 catalog_manager.cc:4841] TS 19a333043931427d8c586d597c30ff8b (127.12.8.1:44217): tablet af19888f835243ed92a9c340c84c6510 (table test-table [id=6d006e5e11ac494daf3794704d6e4b92]) successfully deleted
I20260925 15:15:03.687438 24131 tablet_replica.cc:330] T af19888f835243ed92a9c340c84c6510 P 0c2446bd030f44098a26eb5bc85b6543: stopping tablet replica
I20260925 15:15:03.687902 24131 raft_consensus.cc:2229] T af19888f835243ed92a9c340c84c6510 P 0c2446bd030f44098a26eb5bc85b6543 [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:15:03.688602 24131 raft_consensus.cc:2258] T af19888f835243ed92a9c340c84c6510 P 0c2446bd030f44098a26eb5bc85b6543 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:15:03.690145 24132 ts_tablet_manager.cc:1918] T af19888f835243ed92a9c340c84c6510 P 0e3a8465c355415faa756af2b0b7f832: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:15:03.690356 24132 log.cc:1196] T af19888f835243ed92a9c340c84c6510 P 0e3a8465c355415faa756af2b0b7f832: Deleting WAL directory at /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/minicluster-data/ts-2/wal/wals/af19888f835243ed92a9c340c84c6510
I20260925 15:15:03.690793 24131 ts_tablet_manager.cc:1905] T af19888f835243ed92a9c340c84c6510 P 0c2446bd030f44098a26eb5bc85b6543: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:15:03.690876 24132 ts_tablet_manager.cc:1939] T af19888f835243ed92a9c340c84c6510 P 0e3a8465c355415faa756af2b0b7f832: Deleting consensus metadata
I20260925 15:15:03.692513 23582 catalog_manager.cc:4841] TS 0e3a8465c355415faa756af2b0b7f832 (127.12.8.3:41229): tablet af19888f835243ed92a9c340c84c6510 (table test-table [id=6d006e5e11ac494daf3794704d6e4b92]) successfully deleted
I20260925 15:15:03.696705 24131 ts_tablet_manager.cc:1918] T af19888f835243ed92a9c340c84c6510 P 0c2446bd030f44098a26eb5bc85b6543: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:15:03.696872 24131 log.cc:1196] T af19888f835243ed92a9c340c84c6510 P 0c2446bd030f44098a26eb5bc85b6543: Deleting WAL directory at /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349162370577-12320-0/minicluster-data/ts-1/wal/wals/af19888f835243ed92a9c340c84c6510
I20260925 15:15:03.697299 24131 ts_tablet_manager.cc:1939] T af19888f835243ed92a9c340c84c6510 P 0c2446bd030f44098a26eb5bc85b6543: Deleting consensus metadata
Sep 25 15:15:03 dist-test-slave-p468 krb5kdc[23122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.1@KRBTEST.COM
I20260925 15:15:03.698801 23585 catalog_manager.cc:4841] TS 0c2446bd030f44098a26eb5bc85b6543 (127.12.8.2:46251): tablet af19888f835243ed92a9c340c84c6510 (table test-table [id=6d006e5e11ac494daf3794704d6e4b92]) successfully deleted
I20260925 15:15:03.704764 24103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:03.673729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39110 (local address 127.12.8.1:44217)
0925 15:15:03.673925 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:03.673931 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:03.686736 (+ 12805us) server_negotiation.cc:407] Connection header received
0925 15:15:03.686981 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:03.686987 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:03.687067 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:03.687161 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:03.689059 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:03.690317 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:03.692844 (+ 2527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:03.693479 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:03.693604 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:03.699131 (+ 5527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:03.699143 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:03.699157 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:03.699192 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:03.701955 (+ 2763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:03.702755 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:03.702762 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:03.702764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:03.702883 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:03.703252 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:03.703255 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:03.703257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:03.703611 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:03.703905 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:03.704210 (+ 305us) server_negotiation.cc:299] Negotiation successful
0925 15:15:03.704409 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 15:15:03.705516 23806 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:15:03.795363 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 23678
I20260925 15:15:03.812021 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 23817
Sep 25 15:15:03 dist-test-slave-p468 krb5kdc[23122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.2@KRBTEST.COM
I20260925 15:15:03.831982 24104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:03.813066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46340 (local address 127.12.8.2:46251)
0925 15:15:03.813251 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:03.813260 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:03.813279 (+ 19us) server_negotiation.cc:407] Connection header received
0925 15:15:03.813338 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:03.813343 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:03.813418 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:03.813525 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:03.815176 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:03.816473 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:03.818349 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:03.819021 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:03.819157 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:03.824745 (+ 5588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:03.824757 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:03.824781 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:03.824831 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:03.828906 (+ 4075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:03.829736 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:03.829743 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:03.829745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:03.829876 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:03.830330 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:03.830333 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:03.830335 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:03.830712 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:03.830933 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:03.831536 (+ 603us) server_negotiation.cc:299] Negotiation successful
0925 15:15:03.831698 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 15:15:03.832711 23949 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:15:03.949008 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 23817
I20260925 15:15:03.966701 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 23960
Sep 25 15:15:03 dist-test-slave-p468 krb5kdc[23122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.3@KRBTEST.COM
I20260925 15:15:03.982326 24106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:03.967514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49464 (local address 127.12.8.3:41229)
0925 15:15:03.967672 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:03.967676 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:03.967713 (+ 37us) server_negotiation.cc:407] Connection header received
0925 15:15:03.967893 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:03.967899 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:03.967967 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:03.968064 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:03.969335 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:03.970173 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:03.971507 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:03.971975 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:03.972079 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:03.976492 (+ 4413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:03.976505 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:03.976520 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:03.976554 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:03.979516 (+ 2962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:03.980150 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:03.980157 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:03.980160 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:03.980281 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:03.980673 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:03.980677 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:03.980679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:03.981102 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:03.981493 (+ 391us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:03.981821 (+ 328us) server_negotiation.cc:299] Negotiation successful
0925 15:15:03.981996 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 15:15:03.983035 24083 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:15:04.071403 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 23960
I20260925 15:15:04.087474 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 23566
I20260925 15:15:04.088716 23628 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:15:04.202658 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 23566
I20260925 15:15:04.225389 12320 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 15:15:04.560436 12320 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 15:15:04.560632 12320 mini_ranger.cc:66] Stopping Ranger...
I20260925 15:15:04.905298 12320 mini_ranger.cc:68] Stopped Ranger
2026-09-25 15:15:04.905 UTC [23152] LOG: received smart shutdown request
2026-09-25 15:15:04.909 UTC [23152] LOG: background worker "logical replication launcher" (PID 23161) exited with exit code 1
2026-09-25 15:15:04.909 UTC [23156] LOG: shutting down
2026-09-25 15:15:05.080 UTC [23152] LOG: database system is shut down
2026-09-25T15:15:05Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (48063 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:15:05 dist-test-slave-p468 krb5kdc[24146](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:15:05 dist-test-slave-p468 krb5kdc[24146](info): set up 2 sockets
Sep 25 15:15:05 dist-test-slave-p468 krb5kdc[24146](info): commencing operation
krb5kdc: starting...
W20260925 15:15:06.302559 12320 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/krb5kdc/test-user.keytab.
Sep 25 15:15:06 dist-test-slave-p468 krb5kdc[24146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:15:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:15:06Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.12.8.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.12.8.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.12.8.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/krb5kdc/rangeradmin_127.12.8.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.12.8.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/krb5kdc/rangeradmin_127.12.8.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.12.8.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.12.8.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.12.8.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/krb5kdc/rangerlookup_127.12.8.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.12.8.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/krb5kdc/rangerlookup_127.12.8.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.12.8.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.8.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.8.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/krb5kdc/HTTP_127.12.8.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.12.8.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/krb5kdc/HTTP_127.12.8.20@KRBTEST.COM.keytab.
I20260925 15:15:06.623449 12320 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskBtce3V/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/postgres -l logfile start
2026-09-25 15:15:09.156 UTC [24176] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 15:15:09.157 UTC [24176] LOG: listening on IPv4 address "127.12.8.20", port 44295
2026-09-25 15:15:09.160 UTC [24176] LOG: listening on Unix socket "/tmp/.s.PGSQL.44295"
2026-09-25 15:15:09.185 UTC [24179] LOG: database system was shut down at 2026-09-25 15:15:07 UTC
2026-09-25 15:15:09.189 UTC [24176] LOG: database system is ready to accept connections
I20260925 15:15:10.158317 12320 mini_postgres.cc:96] Postgres bound to 44295
2026-09-25 15:15:10.179 UTC [24187] FATAL: database "slave" does not exist
127.12.8.20:44295 - accepting connections
I20260925 15:15:10.179782 12320 mini_ranger.cc:161] Starting Ranger...
I20260925 15:15:10.204505 12320 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 15:15:10.707371 12320 mini_ranger.cc:87] Created ranger Postgres database
I20260925 15:15:10.707473 12320 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ranger-admin
2026-09-25 15:15:10,996 [I] DB FLAVOR :POSTGRES
2026-09-25 15:15:10,996 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 15:15:10,996 [I] Checking connection..
2026-09-25 15:15:10,996 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBtce3V/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBtce3V/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.8.20:44295/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 15:15:11,256 [I] Checking connection passed.
2026-09-25 15:15:11,257 [I] --------- Verifying version history table ---------
2026-09-25 15:15:11,257 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBtce3V/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBtce3V/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.8.20:44295/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 15:15:11,504 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 15:15:11,505 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBtce3V/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBtce3V/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.8.20:44295/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 15:15:11,753 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 15:15:11,753 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 15:15:11,753 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBtce3V/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBtce3V/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.8.20:44295/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskBtce3V/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 15:15:11.989 UTC [24251] WARNING: there is no transaction in progress
2026-09-25 15:15:11,995 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 15:15:11,995 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBtce3V/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBtce3V/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.8.20:44295/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 15:15:12,256 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 15:15:12,256 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 15:15:12,257 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBtce3V/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBtce3V/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.8.20:44295/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 15:15:12,493 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBtce3V/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBtce3V/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.8.20:44295/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 15:15:12,750 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBtce3V/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBtce3V/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.8.20:44295/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-p468.c.gcp-upstream.internal','N') ;"
2026-09-25 15:15:12,980 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBtce3V/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBtce3V/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.8.20:44295/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 15:15:13,245 [I] Table x_portal_user does not exist in database ranger
2026-09-25 15:15:13,245 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBtce3V/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBtce3V/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.8.20:44295/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 15:15:13,486 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 15:15:13,486 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBtce3V/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBtce3V/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.8.20:44295/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 15:15:13,732 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBtce3V/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBtce3V/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.8.20:44295/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 15:15:13,995 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 15:15:13,996 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBtce3V/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBtce3V/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.8.20:44295/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskBtce3V/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 15:15:14.961 UTC [24359] WARNING: there is no transaction in progress
2026-09-25 15:15:14.977 UTC [24359] WARNING: there is no transaction in progress
2026-09-25 15:15:14.993 UTC [24359] WARNING: there is no transaction in progress
2026-09-25 15:15:15.007 UTC [24359] WARNING: there is no transaction in progress
2026-09-25 15:15:15.023 UTC [24359] WARNING: there is no transaction in progress
2026-09-25 15:15:15.039 UTC [24359] WARNING: there is no transaction in progress
2026-09-25 15:15:15.092 UTC [24359] WARNING: there is no transaction in progress
2026-09-25 15:15:15.098 UTC [24359] WARNING: there is no transaction in progress
2026-09-25 15:15:15.112 UTC [24359] WARNING: there is no transaction in progress
2026-09-25 15:15:15.122 UTC [24359] WARNING: there is no transaction in progress
2026-09-25 15:15:15.132 UTC [24359] WARNING: there is no transaction in progress
2026-09-25 15:15:15.145 UTC [24359] WARNING: there is no transaction in progress
2026-09-25 15:15:15.155 UTC [24359] WARNING: there is no transaction in progress
2026-09-25 15:15:15.161 UTC [24359] WARNING: there is no transaction in progress
2026-09-25 15:15:15.167 UTC [24359] WARNING: there is no transaction in progress
2026-09-25 15:15:15.173 UTC [24359] WARNING: there is no transaction in progress
2026-09-25 15:15:15,781 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 15:15:15,781 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBtce3V/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBtce3V/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.8.20:44295/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 15:15:16,034 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 15:15:16,034 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBtce3V/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBtce3V/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.8.20:44295/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 15:15:16,293 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 15:15:16,293 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBtce3V/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBtce3V/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.8.20:44295/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 15:15:16,525 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 15:15:16,525 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBtce3V/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBtce3V/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.8.20:44295/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 15:15:16,777 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBtce3V/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBtce3V/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.8.20:44295/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 15:15:17,028 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBtce3V/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBtce3V/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.8.20:44295/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-p468.c.gcp-upstream.internal';"
2026-09-25 15:15:17,267 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 15:15:17,267 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBtce3V/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBtce3V/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.8.20:44295/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 15:15:17,502 [I] DB_PATCHES have already been applied
I20260925 15:15:17.507730 12320 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ranger-admin:/tmp/dist-test-taskBtce3V/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskBtce3V/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskBtce3V/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBtce3V/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 15:15:17.507848 12320 mini_ranger.cc:193] Using host: 127.12.8.20
I20260925 15:15:17.522617 12320 mini_ranger.cc:227] Ranger admin URL: http://127.12.8.20:48967
Sep 25, 2026 3:15:17 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 3:15:17 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 3:15:17 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 3:15:17 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 3:15:17 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 3:15:18 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 3:15:18 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 15:15:18.725667 12320 mini_ranger.cc:160] Time spent starting Ranger: real 8.546s user 0.002s sys 0.036s
Sep 25, 2026 3:15:38 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 3:15:38 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 25, 2026 3:15:38 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 3:15:38 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 3:15:38 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 3:15:38 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 3:15:38 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 3:15:38 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 3:15:38 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 3:15:38 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 3:15:38 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 3:15:38 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 3:15:38 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 3:15:38 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 3:15:38 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 3:15:38 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 3:15:38 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 3:15:39 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 15:15:40.335399 12320 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.12.8.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.12.8.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.12.8.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/krb5kdc/rangerkms_127.12.8.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.12.8.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/krb5kdc/rangerkms_127.12.8.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.12.8.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.12.8.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.8.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/krb5kdc/HTTP_127.12.8.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.12.8.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/krb5kdc/HTTP_127.12.8.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 15:15:40 dist-test-slave-p468 krb5kdc[24146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349340, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 15:15:40.495376 12320 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 15:15:40.515969 12320 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 15:15:41.417085 12320 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 15:15:41.417277 12320 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ranger-kms
I20260925 15:15:41.417327 12320 mini_ranger_kms.cc:227] Using postgres at 127.12.8.20:44295
2026-09-25 15:15:41,770 [I] DB FLAVOR :POSTGRES
2026-09-25 15:15:41,770 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 15:15:41,770 [I] Checking connection
2026-09-25 15:15:41,770 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBtce3V/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.8.20:44295/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 15:15:42,011 [I] connection success
2026-09-25 15:15:42,012 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 15:15:42,012 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 15:15:42,012 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBtce3V/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.8.20:44295/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 15:15:42,235 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 15:15:42,235 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 15:15:42,235 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 15:15:42,235 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBtce3V/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.8.20:44295/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 15:15:42,498 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 15:15:42.975723 12320 mini_ranger_kms.cc:310] Created kms service
I20260925 15:15:43.167778 12320 mini_ranger_kms.cc:326] Created kudu user
I20260925 15:15:43.217048 12320 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 15:15:43 dist-test-slave-p468 krb5kdc[24146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349340, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.12.8.20@KRBTEST.COM
I20260925 15:15:43.602144 12320 mini_ranger_kms.cc:384] Added ranger policy
I20260925 15:15:43.602478 12320 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ranger-kms:/tmp/dist-test-taskBtce3V/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBtce3V/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskBtce3V/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 15:15:43.602541 12320 mini_ranger_kms.cc:240] Using host: 127.12.8.20
I20260925 15:15:43.619620 12320 mini_ranger_kms.cc:276] Ranger KMS PID: 24539
I20260925 15:15:43.619746 12320 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.12.8.20:47783
Sep 25, 2026 3:15:44 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 3:15:44 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 3:15:44 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 3:15:44 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 3:15:45 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 3:15:45 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260925 15:15:45.950136 12320 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.455s user 0.063s sys 0.305s
I20260925 15:15:45.950410 12320 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 15:15:45.950577 12320 mini_ranger_kms.cc:401] 127.12.8.20:47783/kms/v1/keys
Sep 25 15:15:46 dist-test-slave-p468 krb5kdc[24146](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790349346, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.8.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 25 15:15:46 dist-test-slave-p468 krb5kdc[24146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349346, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.8.20@KRBTEST.COM for HTTP/127.12.8.20@KRBTEST.COM
Sep 25 15:15:46 dist-test-slave-p468 krb5kdc[24146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349346, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.8.20@KRBTEST.COM for HTTP/127.12.8.20@KRBTEST.COM
Sep 25 15:15:49 dist-test-slave-p468 krb5kdc[24146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349349, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.12.8.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab.
Entry for principal kudu/127.12.8.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab.
I20260925 15:15:49.174459 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.62:36719
--webserver_interface=127.12.8.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab
--principal=kudu/127.12.8.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:44633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.62:36719
--ranger_config_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.8.20:47783/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:15:49.494400 24590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:15:49.494696 24590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:15:49.494782 24590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:15:49.501719 24590 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:15:49.501822 24590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:15:49.501868 24590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:15:49.501924 24590 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:15:49.501971 24590 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:15:49.511574 24590 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:44633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/master-0/wal
--ranger_kms_url=127.12.8.20:47783/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.12.8.62:36719
--ranger_config_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab
--principal=kudu/127.12.8.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.62:36719
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.8.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:15:49.512691 24590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:15:49.513895 24590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:15:49.525544 24595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:49.525590 24598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:49.525504 24596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:15:49.525832 24590 server_base.cc:1034] running on GCE node
I20260925 15:15:49.526779 24590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:15:49.527972 24590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:15:49.529202 24590 hybrid_clock.cc:648] HybridClock initialized: now 1790349349529138 us; error 78 us; skew 500 ppm
I20260925 15:15:49.529534 24590 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:15:49 dist-test-slave-p468 krb5kdc[24146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349349, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:15:49.532583 24590 init.cc:377] Logged in from keytab as kudu/127.12.8.62@KRBTEST.COM (short username kudu)
I20260925 15:15:49.533984 24590 webserver.cc:460] Webserver started at http://127.12.8.62:37713/ using document root <none> and password file <none>
I20260925 15:15:49.534655 24590 fs_manager.cc:359] Metadata directory not provided
I20260925 15:15:49.534789 24590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:15:49.535082 24590 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 15:15:49 dist-test-slave-p468 krb5kdc[24146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349349, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.62@KRBTEST.COM for HTTP/127.12.8.20@KRBTEST.COM
I20260925 15:15:49.582135 24590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/master-0/data/instance:
uuid: "68b74fef56d747d6a6f907ef1c9f83dd"
format_stamp: "Formatted at 2026-09-25 15:15:49 on dist-test-slave-p468"
server_key: "6d74649856559c91b6448a983b444ff2"
server_key_iv: "9cdff1edc57395cbca7a6019ebd8987e"
server_key_version: "kuduclusterkey@0"
I20260925 15:15:49.583071 24590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/master-0/wal/instance:
uuid: "68b74fef56d747d6a6f907ef1c9f83dd"
format_stamp: "Formatted at 2026-09-25 15:15:49 on dist-test-slave-p468"
server_key: "6d74649856559c91b6448a983b444ff2"
server_key_iv: "9cdff1edc57395cbca7a6019ebd8987e"
server_key_version: "kuduclusterkey@0"
I20260925 15:15:49.588507 24590 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 15:15:49.604796 24605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:49.606446 24590 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:15:49.606640 24590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/master-0/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/master-0/wal
uuid: "68b74fef56d747d6a6f907ef1c9f83dd"
format_stamp: "Formatted at 2026-09-25 15:15:49 on dist-test-slave-p468"
server_key: "6d74649856559c91b6448a983b444ff2"
server_key_iv: "9cdff1edc57395cbca7a6019ebd8987e"
server_key_version: "kuduclusterkey@0"
I20260925 15:15:49.606869 24590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:15:49.622699 24590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:15:49.626344 24590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:15:49.626655 24590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:15:49.645045 24590 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.62:36719
I20260925 15:15:49.645078 24656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.62:36719 every 8 connection(s)
I20260925 15:15:49.646497 24590 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
I20260925 15:15:49.650202 24657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:15:49.656248 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 24590
I20260925 15:15:49.656482 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/master-0/wal/instance
I20260925 15:15:49.657353 12320 external_mini_cluster.cc:1428] Setting key 475e4eb27c7fb6bb9c6ea0b2116e65d8
I20260925 15:15:49.661414 24657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68b74fef56d747d6a6f907ef1c9f83dd: Bootstrap starting.
I20260925 15:15:49.664506 24657 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 68b74fef56d747d6a6f907ef1c9f83dd: Neither blocks nor log segments found. Creating new log.
I20260925 15:15:49.665534 24657 log.cc:824] T 00000000000000000000000000000000 P 68b74fef56d747d6a6f907ef1c9f83dd: Log is configured to *not* fsync() on all Append() calls
I20260925 15:15:49.668269 24657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68b74fef56d747d6a6f907ef1c9f83dd: No bootstrap required, opened a new log
Sep 25 15:15:49 dist-test-slave-p468 krb5kdc[24146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349349, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
I20260925 15:15:49.677060 24657 raft_consensus.cc:348] T 00000000000000000000000000000000 P 68b74fef56d747d6a6f907ef1c9f83dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68b74fef56d747d6a6f907ef1c9f83dd" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 36719 } }
I20260925 15:15:49.677460 24657 raft_consensus.cc:374] T 00000000000000000000000000000000 P 68b74fef56d747d6a6f907ef1c9f83dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:15:49.677600 24657 raft_consensus.cc:729] T 00000000000000000000000000000000 P 68b74fef56d747d6a6f907ef1c9f83dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68b74fef56d747d6a6f907ef1c9f83dd, State: Initialized, Role: FOLLOWER
I20260925 15:15:49.678365 24657 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68b74fef56d747d6a6f907ef1c9f83dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68b74fef56d747d6a6f907ef1c9f83dd" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 36719 } }
I20260925 15:15:49.678658 24657 raft_consensus.cc:388] T 00000000000000000000000000000000 P 68b74fef56d747d6a6f907ef1c9f83dd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:15:49.678819 24657 raft_consensus.cc:482] T 00000000000000000000000000000000 P 68b74fef56d747d6a6f907ef1c9f83dd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:15:49.679023 24657 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 68b74fef56d747d6a6f907ef1c9f83dd [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:15:49.679930 24660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:49.659580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59598 (local address 127.12.8.62:36719)
0925 15:15:49.660333 (+ 753us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:49.660344 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:49.660384 (+ 40us) server_negotiation.cc:407] Connection header received
0925 15:15:49.662117 (+ 1733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:49.662142 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:49.662576 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:49.662978 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:49.664359 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:49.665773 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:49.667199 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:49.667888 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:49.668078 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:49.672158 (+ 4080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:49.672165 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:49.672183 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:49.672224 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:49.675481 (+ 3257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:49.676182 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:49.676194 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:49.676200 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:49.676344 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:49.676712 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:49.676719 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:49.676722 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:49.677229 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:49.677530 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:49.677982 (+ 452us) server_negotiation.cc:299] Negotiation successful
0925 15:15:49.678437 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":347,"threads_started":1}
I20260925 15:15:49.681813 24657 raft_consensus.cc:504] T 00000000000000000000000000000000 P 68b74fef56d747d6a6f907ef1c9f83dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68b74fef56d747d6a6f907ef1c9f83dd" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 36719 } }
I20260925 15:15:49.682401 24657 leader_election.cc:302] T 00000000000000000000000000000000 P 68b74fef56d747d6a6f907ef1c9f83dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 68b74fef56d747d6a6f907ef1c9f83dd; no voters:
I20260925 15:15:49.683027 24657 leader_election.cc:288] T 00000000000000000000000000000000 P 68b74fef56d747d6a6f907ef1c9f83dd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:15:49.683135 24662 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 68b74fef56d747d6a6f907ef1c9f83dd [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:15:49.683845 24662 raft_consensus.cc:686] T 00000000000000000000000000000000 P 68b74fef56d747d6a6f907ef1c9f83dd [term 1 LEADER]: Becoming Leader. State: Replica: 68b74fef56d747d6a6f907ef1c9f83dd, State: Running, Role: LEADER
I20260925 15:15:49.684355 24662 consensus_queue.cc:237] T 00000000000000000000000000000000 P 68b74fef56d747d6a6f907ef1c9f83dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68b74fef56d747d6a6f907ef1c9f83dd" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 36719 } }
I20260925 15:15:49.685268 24657 sys_catalog.cc:564] T 00000000000000000000000000000000 P 68b74fef56d747d6a6f907ef1c9f83dd [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:15:49.687827 24663 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68b74fef56d747d6a6f907ef1c9f83dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "68b74fef56d747d6a6f907ef1c9f83dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68b74fef56d747d6a6f907ef1c9f83dd" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 36719 } } }
I20260925 15:15:49.687821 24664 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68b74fef56d747d6a6f907ef1c9f83dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68b74fef56d747d6a6f907ef1c9f83dd. Latest consensus state: current_term: 1 leader_uuid: "68b74fef56d747d6a6f907ef1c9f83dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68b74fef56d747d6a6f907ef1c9f83dd" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 36719 } } }
I20260925 15:15:49.688230 24663 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68b74fef56d747d6a6f907ef1c9f83dd [sys.catalog]: This master's current role is: LEADER
I20260925 15:15:49.688294 24664 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68b74fef56d747d6a6f907ef1c9f83dd [sys.catalog]: This master's current role is: LEADER
I20260925 15:15:49.695215 24668 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:15:49.699990 24668 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:15:49.703920 24657 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 15:15:49.718043 24668 catalog_manager.cc:1347] Generated new cluster ID: eaff2b2162c74873b638de4828f87f77
I20260925 15:15:49.718219 24668 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:15:49.731277 24668 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:15:49.733248 24668 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:15:49.752616 24668 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 68b74fef56d747d6a6f907ef1c9f83dd: Generated new TSK 0
I20260925 15:15:49.753834 24668 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.8.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.1.keytab.
Entry for principal kudu/127.12.8.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.1.keytab.
I20260925 15:15:50.491953 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.8.20:47783/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.1:0
--local_ip_for_outbound_sockets=127.12.8.1
--webserver_interface=127.12.8.1
--webserver_port=0
--tserver_master_addrs=127.12.8.62:36719
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.1.keytab
--principal=kudu/127.12.8.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:44633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 15:15:50 dist-test-slave-p468 krb5kdc[24146](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790349350, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260925 15:15:50.936501 24702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:15:50.936870 24702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:15:50.937077 24702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:15:50.948566 24702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:15:50.948755 24702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:15:50.948969 24702 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.1
I20260925 15:15:50.964313 24702 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:44633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.12.8.20:47783/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.1.keytab
--principal=kudu/127.12.8.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.8.1
--webserver_port=0
--tserver_master_addrs=127.12.8.62:36719
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.1
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:15:50.965967 24702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:15:50.967593 24702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:15:50.983691 24712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:50.985800 24713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:50.992522 24715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:15:50.993463 24702 server_base.cc:1034] running on GCE node
I20260925 15:15:50.994310 24702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:15:51.004388 24702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:15:51.007819 24702 hybrid_clock.cc:648] HybridClock initialized: now 1790349351007702 us; error 123 us; skew 500 ppm
I20260925 15:15:51.008256 24702 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:15:51 dist-test-slave-p468 krb5kdc[24146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349351, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:15:51.012487 24702 init.cc:377] Logged in from keytab as kudu/127.12.8.1@KRBTEST.COM (short username kudu)
I20260925 15:15:51.018595 24702 webserver.cc:460] Webserver started at http://127.12.8.1:42413/ using document root <none> and password file <none>
I20260925 15:15:51.019481 24702 fs_manager.cc:359] Metadata directory not provided
I20260925 15:15:51.019637 24702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:15:51.020009 24702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 15:15:51 dist-test-slave-p468 krb5kdc[24146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349351, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.1@KRBTEST.COM for HTTP/127.12.8.20@KRBTEST.COM
I20260925 15:15:51.040305 24702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ts-0/data/instance:
uuid: "fe814b5e4d664d1793278bd20185f560"
format_stamp: "Formatted at 2026-09-25 15:15:51 on dist-test-slave-p468"
server_key: "2128f202321e060d2a03c3b98cbf10d1"
server_key_iv: "1053c103414276eeaca0a77e9dbaf87e"
server_key_version: "kuduclusterkey@0"
I20260925 15:15:51.041337 24702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ts-0/wal/instance:
uuid: "fe814b5e4d664d1793278bd20185f560"
format_stamp: "Formatted at 2026-09-25 15:15:51 on dist-test-slave-p468"
server_key: "2128f202321e060d2a03c3b98cbf10d1"
server_key_iv: "1053c103414276eeaca0a77e9dbaf87e"
server_key_version: "kuduclusterkey@0"
I20260925 15:15:51.048336 24702 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260925 15:15:51.076673 24722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:51.078935 24702 fs_manager.cc:725] Time spent opening block manager: real 0.015s user 0.007s sys 0.001s
I20260925 15:15:51.079221 24702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ts-0/wal
uuid: "fe814b5e4d664d1793278bd20185f560"
format_stamp: "Formatted at 2026-09-25 15:15:51 on dist-test-slave-p468"
server_key: "2128f202321e060d2a03c3b98cbf10d1"
server_key_iv: "1053c103414276eeaca0a77e9dbaf87e"
server_key_version: "kuduclusterkey@0"
I20260925 15:15:51.079622 24702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:15:51.112504 24702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:15:51.117936 24702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:15:51.118407 24702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:15:51.119798 24702 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:15:51.121716 24702 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:15:51.121827 24702 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:51.121966 24702 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:15:51.122043 24702 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 15:15:51 dist-test-slave-p468 krb5kdc[24146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349350, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.62@KRBTEST.COM for HTTP/127.12.8.20@KRBTEST.COM
I20260925 15:15:51.319559 24834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.1:36817 every 8 connection(s)
I20260925 15:15:51.319967 24702 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.1:36817
I20260925 15:15:51.321806 24702 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ts-0/data/info.pb
I20260925 15:15:51.331629 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 24702
I20260925 15:15:51.331820 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ts-0/wal/instance
I20260925 15:15:51.332551 12320 external_mini_cluster.cc:1428] Setting key 0b02d82818342c270029e993a6953afb
Sep 25 15:15:51 dist-test-slave-p468 krb5kdc[24146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349351, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.1@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.8.2@KRBTEST.COM; defaulting to no policy
I20260925 15:15:51.374857 24837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:51.330522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.1:33169 (local address 127.12.8.62:36719)
0925 15:15:51.352489 (+ 21967us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:51.352496 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:51.352518 (+ 22us) server_negotiation.cc:407] Connection header received
0925 15:15:51.352580 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:51.352585 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:51.352677 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:51.352800 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:51.355236 (+ 2436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:51.356485 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:51.359412 (+ 2927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:51.360278 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:51.360431 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:51.365765 (+ 5334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:51.365775 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:51.365783 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:51.365835 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:51.370416 (+ 4581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:51.371597 (+ 1181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:51.371603 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:51.371607 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:51.371769 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:51.372423 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:51.372430 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:51.372433 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:51.372852 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:51.373022 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:51.374299 (+ 1277us) server_negotiation.cc:299] Negotiation successful
0925 15:15:51.374566 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21823,"thread_start_us":21715,"threads_started":1}
I20260925 15:15:51.376907 24836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:51.329842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:36719 (local address 127.12.8.1:33169)
0925 15:15:51.349174 (+ 19332us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:51.349242 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:51.351749 (+ 2507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:51.353123 (+ 1374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:51.353138 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:51.353847 (+ 709us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:15:51.354963 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:51.354997 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:51.356666 (+ 1669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:51.356675 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:51.359213 (+ 2538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:51.359232 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:51.360771 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:51.360781 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:51.361147 (+ 366us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:51.362001 (+ 854us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:15:51.362048 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:15:51.365458 (+ 3410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:15:51.370638 (+ 5180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:51.370652 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:51.370677 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:51.371415 (+ 738us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:51.371921 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:51.371926 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:51.371931 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:51.372257 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:51.373125 (+ 868us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:15:51.373138 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:15:51.373649 (+ 511us) client_negotiation.cc:769] Sending connection context
0925 15:15:51.373974 (+ 325us) client_negotiation.cc:240] Negotiation successful
0925 15:15:51.375601 (+ 1627us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18961,"thread_start_us":18732,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.2@KRBTEST.COM" created.
I20260925 15:15:51.379073 24835 heartbeater.cc:347] Connected to a master server at 127.12.8.62:36719
I20260925 15:15:51.379402 24835 heartbeater.cc:464] Registering TS with master...
I20260925 15:15:51.380327 24835 heartbeater.cc:511] Master 127.12.8.62:36719 requested a full tablet report, sending...
I20260925 15:15:51.387120 24622 ts_manager.cc:194] Registered new tserver with Master: fe814b5e4d664d1793278bd20185f560 (127.12.8.1:36817)
I20260925 15:15:51.388753 24622 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.1@KRBTEST.COM'} at 127.12.8.1:33169
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.2.keytab.
Entry for principal kudu/127.12.8.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.2.keytab.
I20260925 15:15:51.413383 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.8.20:47783/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.2:0
--local_ip_for_outbound_sockets=127.12.8.2
--webserver_interface=127.12.8.2
--webserver_port=0
--tserver_master_addrs=127.12.8.62:36719
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.2.keytab
--principal=kudu/127.12.8.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:44633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 15:15:51 dist-test-slave-p468 krb5kdc[24146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349350, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.62@KRBTEST.COM for HTTP/127.12.8.20@KRBTEST.COM
W20260925 15:15:51.863548 24841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:15:51.864043 24841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:15:51.864214 24841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:15:51.871773 24841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:15:51.871923 24841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:15:51.872112 24841 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.2
I20260925 15:15:51.881812 24841 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:44633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.12.8.20:47783/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.2.keytab
--principal=kudu/127.12.8.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.8.2
--webserver_port=0
--tserver_master_addrs=127.12.8.62:36719
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.2
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:15:51.883221 24841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:15:51.884469 24841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:15:51.898644 24855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:51.899269 24856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:51.900163 24858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:15:51.900823 24841 server_base.cc:1034] running on GCE node
I20260925 15:15:51.901460 24841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:15:51.902570 24841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:15:51.903831 24841 hybrid_clock.cc:648] HybridClock initialized: now 1790349351903789 us; error 40 us; skew 500 ppm
I20260925 15:15:51.904152 24841 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:15:51 dist-test-slave-p468 krb5kdc[24146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349351, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:15:51.907634 24841 init.cc:377] Logged in from keytab as kudu/127.12.8.2@KRBTEST.COM (short username kudu)
I20260925 15:15:51.909446 24841 webserver.cc:460] Webserver started at http://127.12.8.2:44265/ using document root <none> and password file <none>
I20260925 15:15:51.910357 24841 fs_manager.cc:359] Metadata directory not provided
I20260925 15:15:51.910513 24841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:15:51.910902 24841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 15:15:51 dist-test-slave-p468 krb5kdc[24146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349351, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.2@KRBTEST.COM for HTTP/127.12.8.20@KRBTEST.COM
I20260925 15:15:51.929632 24841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ts-1/data/instance:
uuid: "282ae5bf6e4142d5a298b28ca63b32b0"
format_stamp: "Formatted at 2026-09-25 15:15:51 on dist-test-slave-p468"
server_key: "38fe1e0da2d3bd68a8337c696a6a5105"
server_key_iv: "bccd8cc886b03db2fe9fb4c046d6764d"
server_key_version: "kuduclusterkey@0"
I20260925 15:15:51.930752 24841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ts-1/wal/instance:
uuid: "282ae5bf6e4142d5a298b28ca63b32b0"
format_stamp: "Formatted at 2026-09-25 15:15:51 on dist-test-slave-p468"
server_key: "38fe1e0da2d3bd68a8337c696a6a5105"
server_key_iv: "bccd8cc886b03db2fe9fb4c046d6764d"
server_key_version: "kuduclusterkey@0"
I20260925 15:15:51.938225 24841 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.004s
I20260925 15:15:51.958709 24865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:51.960590 24841 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260925 15:15:51.960819 24841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ts-1/wal
uuid: "282ae5bf6e4142d5a298b28ca63b32b0"
format_stamp: "Formatted at 2026-09-25 15:15:51 on dist-test-slave-p468"
server_key: "38fe1e0da2d3bd68a8337c696a6a5105"
server_key_iv: "bccd8cc886b03db2fe9fb4c046d6764d"
server_key_version: "kuduclusterkey@0"
I20260925 15:15:51.961125 24841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:15:51.997170 24841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:15:52.002833 24841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:15:52.003254 24841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:15:52.004752 24841 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:15:52.006937 24841 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:15:52.007072 24841 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260925 15:15:52.007266 24841 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:15:52.007359 24841 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:52.054116 24841 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.2:43965
I20260925 15:15:52.054168 24977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.2:43965 every 8 connection(s)
I20260925 15:15:52.055932 24841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ts-1/data/info.pb
I20260925 15:15:52.058388 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 24841
I20260925 15:15:52.058571 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ts-1/wal/instance
I20260925 15:15:52.059149 12320 external_mini_cluster.cc:1428] Setting key 12d4342788f997428219564340407b2f
Sep 25 15:15:52 dist-test-slave-p468 krb5kdc[24146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349351, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.2@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.8.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.3@KRBTEST.COM" created.
I20260925 15:15:52.099972 24981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:52.061953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:36719 (local address 127.12.8.2:53533)
0925 15:15:52.070805 (+ 8852us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:52.070860 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:52.073218 (+ 2358us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:52.079932 (+ 6714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:52.079945 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:52.080592 (+ 647us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:15:52.081417 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:52.081443 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:52.083013 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:52.083019 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:52.085152 (+ 2133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:52.085170 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:52.086377 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:52.086385 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:52.086722 (+ 337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:52.087569 (+ 847us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:15:52.087605 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:15:52.090626 (+ 3021us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:15:52.094978 (+ 4352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:52.094987 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:52.095005 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:52.095619 (+ 614us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:52.096090 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:52.096096 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:52.096100 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:52.096373 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:52.097127 (+ 754us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:15:52.097136 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:15:52.097575 (+ 439us) client_negotiation.cc:769] Sending connection context
0925 15:15:52.097935 (+ 360us) client_negotiation.cc:240] Negotiation successful
0925 15:15:52.098518 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8547,"thread_start_us":8290,"threads_started":1}
I20260925 15:15:52.101354 24980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:52.061557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.2:53533 (local address 127.12.8.62:36719)
0925 15:15:52.079307 (+ 17750us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:52.079313 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:52.079336 (+ 23us) server_negotiation.cc:407] Connection header received
0925 15:15:52.079418 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:52.079424 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:52.079509 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:52.079642 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:52.081639 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:52.082798 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:52.085375 (+ 2577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:52.086011 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:52.086133 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:52.090943 (+ 4810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:52.090949 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:52.090955 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:52.090996 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:52.094799 (+ 3803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:52.095791 (+ 992us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:52.095796 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:52.095801 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:52.095953 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:52.096518 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:52.096523 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:52.096527 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:52.096907 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:52.097043 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:52.100711 (+ 3668us) server_negotiation.cc:299] Negotiation successful
0925 15:15:52.100960 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17597,"thread_start_us":18191,"threads_started":1}
I20260925 15:15:52.102618 24978 heartbeater.cc:347] Connected to a master server at 127.12.8.62:36719
I20260925 15:15:52.103003 24978 heartbeater.cc:464] Registering TS with master...
I20260925 15:15:52.103860 24978 heartbeater.cc:511] Master 127.12.8.62:36719 requested a full tablet report, sending...
I20260925 15:15:52.105545 24622 ts_manager.cc:194] Registered new tserver with Master: 282ae5bf6e4142d5a298b28ca63b32b0 (127.12.8.2:43965)
I20260925 15:15:52.106773 24622 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.2@KRBTEST.COM'} at 127.12.8.2:53533
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.3.keytab.
Entry for principal kudu/127.12.8.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.3.keytab.
I20260925 15:15:52.134784 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.8.20:47783/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.3:0
--local_ip_for_outbound_sockets=127.12.8.3
--webserver_interface=127.12.8.3
--webserver_port=0
--tserver_master_addrs=127.12.8.62:36719
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.3.keytab
--principal=kudu/127.12.8.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:44633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:15:52.392758 24835 heartbeater.cc:503] Master 127.12.8.62:36719 was elected leader, sending a full tablet report...
W20260925 15:15:52.494768 24984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:15:52.495023 24984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:15:52.495114 24984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:15:52.502317 24984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:15:52.502411 24984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:15:52.502519 24984 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.3
I20260925 15:15:52.512020 24984 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:44633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.12.8.20:47783/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.3.keytab
--principal=kudu/127.12.8.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.8.3
--webserver_port=0
--tserver_master_addrs=127.12.8.62:36719
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.3
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:15:52.513197 24984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:15:52.514406 24984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:15:52.527029 24989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:52.527098 24990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:52.527690 24992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:15:52.527976 24984 server_base.cc:1034] running on GCE node
I20260925 15:15:52.528520 24984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:15:52.529475 24984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:15:52.530725 24984 hybrid_clock.cc:648] HybridClock initialized: now 1790349352530687 us; error 51 us; skew 500 ppm
I20260925 15:15:52.531036 24984 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:15:52 dist-test-slave-p468 krb5kdc[24146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349352, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:15:52.534135 24984 init.cc:377] Logged in from keytab as kudu/127.12.8.3@KRBTEST.COM (short username kudu)
I20260925 15:15:52.535696 24984 webserver.cc:460] Webserver started at http://127.12.8.3:37547/ using document root <none> and password file <none>
I20260925 15:15:52.536350 24984 fs_manager.cc:359] Metadata directory not provided
I20260925 15:15:52.536491 24984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:15:52.536798 24984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 15:15:52 dist-test-slave-p468 krb5kdc[24146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349352, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.3@KRBTEST.COM for HTTP/127.12.8.20@KRBTEST.COM
I20260925 15:15:52.550953 24984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ts-2/data/instance:
uuid: "e6072e984cd04509a1b66632b5638eed"
format_stamp: "Formatted at 2026-09-25 15:15:52 on dist-test-slave-p468"
server_key: "3b533adc0f61b27007b3dc8a85d6e01e"
server_key_iv: "d22283fd65ff9c446a952fd314bf7760"
server_key_version: "kuduclusterkey@0"
I20260925 15:15:52.551915 24984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ts-2/wal/instance:
uuid: "e6072e984cd04509a1b66632b5638eed"
format_stamp: "Formatted at 2026-09-25 15:15:52 on dist-test-slave-p468"
server_key: "3b533adc0f61b27007b3dc8a85d6e01e"
server_key_iv: "d22283fd65ff9c446a952fd314bf7760"
server_key_version: "kuduclusterkey@0"
I20260925 15:15:52.557276 24984 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 15:15:52.571098 24999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:52.572702 24984 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:15:52.572865 24984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ts-2/wal
uuid: "e6072e984cd04509a1b66632b5638eed"
format_stamp: "Formatted at 2026-09-25 15:15:52 on dist-test-slave-p468"
server_key: "3b533adc0f61b27007b3dc8a85d6e01e"
server_key_iv: "d22283fd65ff9c446a952fd314bf7760"
server_key_version: "kuduclusterkey@0"
I20260925 15:15:52.573067 24984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:15:52.625785 24984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:15:52.629519 24984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:15:52.629846 24984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:15:52.630842 24984 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:15:52.632184 24984 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:15:52.632256 24984 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:52.632417 24984 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:15:52.632468 24984 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:52.666282 24984 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.3:36233
I20260925 15:15:52.666376 25111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.3:36233 every 8 connection(s)
I20260925 15:15:52.667722 24984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ts-2/data/info.pb
I20260925 15:15:52.674259 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 24984
I20260925 15:15:52.674437 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ts-2/wal/instance
I20260925 15:15:52.674953 12320 external_mini_cluster.cc:1428] Setting key 117910f6254b985a2d99f6a0affcca34
Sep 25 15:15:52 dist-test-slave-p468 krb5kdc[24146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349352, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.3@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
I20260925 15:15:52.687310 25114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:52.670240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.3:53529 (local address 127.12.8.62:36719)
0925 15:15:52.670706 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:52.670711 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:52.671644 (+ 933us) server_negotiation.cc:407] Connection header received
0925 15:15:52.673380 (+ 1736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:52.673385 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:52.673449 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:52.673546 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:52.675169 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:52.676127 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:52.677942 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:52.678506 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:52.678610 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:52.681628 (+ 3018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:52.681633 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:52.681636 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:52.681675 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:52.684480 (+ 2805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:52.685219 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:52.685223 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:52.685225 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:52.685346 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:52.685701 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:52.685704 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:52.685707 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:52.686042 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:52.686172 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:52.686865 (+ 693us) server_negotiation.cc:299] Negotiation successful
0925 15:15:52.687030 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":218,"threads_started":1}
I20260925 15:15:52.688220 25115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:52.670822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:36719 (local address 127.12.8.3:53529)
0925 15:15:52.671496 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:52.671531 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:52.673132 (+ 1601us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:52.673802 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:52.673810 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:52.674291 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:15:52.674965 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:52.674982 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:52.676289 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:52.676297 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:52.677804 (+ 1507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:52.677821 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:52.678705 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:52.678710 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:52.678931 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:52.679499 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:15:52.679528 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:15:52.681427 (+ 1899us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:15:52.684604 (+ 3177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:52.684610 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:52.684620 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:52.685091 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:52.685443 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:52.685446 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:52.685448 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:52.685605 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:52.686164 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:15:52.686169 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:15:52.686503 (+ 334us) client_negotiation.cc:769] Sending connection context
0925 15:15:52.686751 (+ 248us) client_negotiation.cc:240] Negotiation successful
0925 15:15:52.687109 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":286,"threads_started":1}
I20260925 15:15:52.689769 25112 heartbeater.cc:347] Connected to a master server at 127.12.8.62:36719
I20260925 15:15:52.690157 25112 heartbeater.cc:464] Registering TS with master...
I20260925 15:15:52.691069 25112 heartbeater.cc:511] Master 127.12.8.62:36719 requested a full tablet report, sending...
I20260925 15:15:52.692363 24621 ts_manager.cc:194] Registered new tserver with Master: e6072e984cd04509a1b66632b5638eed (127.12.8.3:36233)
I20260925 15:15:52.693274 24621 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.3@KRBTEST.COM'} at 127.12.8.3:53529
I20260925 15:15:52.703539 12320 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:15:52.705281 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 24702
Sep 25 15:15:52 dist-test-slave-p468 krb5kdc[24146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349349, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.1@KRBTEST.COM
I20260925 15:15:52.724679 25117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:52.706305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47664 (local address 127.12.8.1:36817)
0925 15:15:52.710318 (+ 4013us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:52.710327 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:52.710360 (+ 33us) server_negotiation.cc:407] Connection header received
0925 15:15:52.710416 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:52.710422 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:52.710635 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:52.710809 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:52.712138 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:52.713130 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:52.714653 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:52.715166 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:52.715276 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:52.719296 (+ 4020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:52.719306 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:52.719320 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:52.719354 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:52.722150 (+ 2796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:52.722834 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:52.722841 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:52.722843 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:52.722986 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:52.723313 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:52.723316 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:52.723318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:52.723635 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:52.723865 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:52.724192 (+ 327us) server_negotiation.cc:299] Negotiation successful
0925 15:15:52.724379 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3798,"thread_start_us":3637,"threads_started":1}
I20260925 15:15:52.725890 24830 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:15:52.808372 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 24702
I20260925 15:15:52.823899 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 24841
Sep 25 15:15:52 dist-test-slave-p468 krb5kdc[24146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349349, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.2@KRBTEST.COM
I20260925 15:15:52.839496 25119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:52.824828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35524 (local address 127.12.8.2:43965)
0925 15:15:52.825415 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:52.825422 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:52.825447 (+ 25us) server_negotiation.cc:407] Connection header received
0925 15:15:52.825506 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:52.825512 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:52.825720 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:52.825859 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:52.827104 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:52.828106 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:52.829318 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:52.829843 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:52.829943 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:52.834021 (+ 4078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:52.834032 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:52.834046 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:52.834077 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:52.837030 (+ 2953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:52.837602 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:52.837608 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:52.837611 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:52.837730 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:52.838046 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:52.838049 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:52.838050 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:52.838434 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:52.838672 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:52.838914 (+ 242us) server_negotiation.cc:299] Negotiation successful
0925 15:15:52.839100 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":300,"threads_started":1}
I20260925 15:15:52.840826 24973 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:15:52.922755 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 24841
I20260925 15:15:52.937974 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 24984
Sep 25 15:15:52 dist-test-slave-p468 krb5kdc[24146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349349, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.3@KRBTEST.COM
I20260925 15:15:52.954614 25121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:52.939062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39328 (local address 127.12.8.3:36233)
0925 15:15:52.939644 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:52.939650 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:52.939679 (+ 29us) server_negotiation.cc:407] Connection header received
0925 15:15:52.939735 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:52.939741 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:52.939945 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:52.940133 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:52.941437 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:52.942668 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:52.944026 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:52.944592 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:52.944693 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:52.949035 (+ 4342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:52.949047 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:52.949060 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:52.949091 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:52.952051 (+ 2960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:52.952753 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:52.952760 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:52.952762 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:52.952886 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:52.953217 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:52.953220 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:52.953222 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:52.953538 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:52.953779 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:52.954059 (+ 280us) server_negotiation.cc:299] Negotiation successful
0925 15:15:52.954318 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":300,"threads_started":1}
I20260925 15:15:52.955880 25107 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:15:53.037554 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 24984
I20260925 15:15:53.053058 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 24590
I20260925 15:15:53.054450 24652 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:15:53.164690 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 24590
I20260925 15:15:53.185864 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.62:36719
--webserver_interface=127.12.8.62
--webserver_port=37713
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab
--principal=kudu/127.12.8.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:44633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.62:36719
--ranger_config_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.8.20:47783/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.12.8.20:47783/kms with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:15:53.502604 25124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:15:53.502879 25124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:15:53.503043 25124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:15:53.510035 25124 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:15:53.510133 25124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:15:53.510229 25124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:15:53.510339 25124 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:15:53.510411 25124 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:15:53.520087 25124 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:44633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.12.8.20:47783/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.12.8.62:36719
--ranger_config_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab
--principal=kudu/127.12.8.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.62:36719
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.8.62
--webserver_port=37713
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:15:53.521302 25124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:15:53.522629 25124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:15:53.534009 25130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:53.534068 25129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:53.533975 25132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:15:53.534933 25124 server_base.cc:1034] running on GCE node
I20260925 15:15:53.535567 25124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:15:53.536856 25124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:15:53.538074 25124 hybrid_clock.cc:648] HybridClock initialized: now 1790349353538017 us; error 46 us; skew 500 ppm
I20260925 15:15:53.538458 25124 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:15:53 dist-test-slave-p468 krb5kdc[24146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349353, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:15:53.541555 25124 init.cc:377] Logged in from keytab as kudu/127.12.8.62@KRBTEST.COM (short username kudu)
I20260925 15:15:53.543100 25124 webserver.cc:460] Webserver started at http://127.12.8.62:37713/ using document root <none> and password file <none>
I20260925 15:15:53.543701 25124 fs_manager.cc:359] Metadata directory not provided
I20260925 15:15:53.543862 25124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 15:15:53 dist-test-slave-p468 krb5kdc[24146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349353, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.62@KRBTEST.COM for HTTP/127.12.8.20@KRBTEST.COM
I20260925 15:15:53.601181 25124 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20260925 15:15:53.604172 25140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:53.605499 25124 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 15:15:53.605691 25124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/master-0/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/master-0/wal
uuid: "68b74fef56d747d6a6f907ef1c9f83dd"
format_stamp: "Formatted at 2026-09-25 15:15:49 on dist-test-slave-p468"
server_key: "6d74649856559c91b6448a983b444ff2"
server_key_iv: "9cdff1edc57395cbca7a6019ebd8987e"
server_key_version: "kuduclusterkey@0"
I20260925 15:15:53.606722 25124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:15:53.625033 25124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:15:53.628818 25124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:15:53.629103 25124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:15:53.647778 25124 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.62:36719
I20260925 15:15:53.647800 25191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.62:36719 every 8 connection(s)
I20260925 15:15:53.649204 25124 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
I20260925 15:15:53.655905 25192 sys_catalog.cc:263] Verifying existing consensus state
I20260925 15:15:53.658133 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 25124
I20260925 15:15:53.659049 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.8.20:47783/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.1:36817
--local_ip_for_outbound_sockets=127.12.8.1
--tserver_master_addrs=127.12.8.62:36719
--webserver_port=42413
--webserver_interface=127.12.8.1
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.1.keytab
--principal=kudu/127.12.8.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:44633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.12.8.20:47783/kms with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:15:53.659116 25192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68b74fef56d747d6a6f907ef1c9f83dd: Bootstrap starting.
I20260925 15:15:53.677456 25192 log.cc:824] T 00000000000000000000000000000000 P 68b74fef56d747d6a6f907ef1c9f83dd: Log is configured to *not* fsync() on all Append() calls
I20260925 15:15:53.686043 25192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68b74fef56d747d6a6f907ef1c9f83dd: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 15:15:53.686614 25192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68b74fef56d747d6a6f907ef1c9f83dd: Bootstrap complete.
I20260925 15:15:53.695446 25192 raft_consensus.cc:348] T 00000000000000000000000000000000 P 68b74fef56d747d6a6f907ef1c9f83dd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68b74fef56d747d6a6f907ef1c9f83dd" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 36719 } }
I20260925 15:15:53.696259 25192 raft_consensus.cc:729] T 00000000000000000000000000000000 P 68b74fef56d747d6a6f907ef1c9f83dd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68b74fef56d747d6a6f907ef1c9f83dd, State: Initialized, Role: FOLLOWER
I20260925 15:15:53.696928 25192 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68b74fef56d747d6a6f907ef1c9f83dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68b74fef56d747d6a6f907ef1c9f83dd" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 36719 } }
I20260925 15:15:53.697148 25192 raft_consensus.cc:388] T 00000000000000000000000000000000 P 68b74fef56d747d6a6f907ef1c9f83dd [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:15:53.697256 25192 raft_consensus.cc:482] T 00000000000000000000000000000000 P 68b74fef56d747d6a6f907ef1c9f83dd [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:15:53.697453 25192 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 68b74fef56d747d6a6f907ef1c9f83dd [term 1 FOLLOWER]: Advancing to term 2
I20260925 15:15:53.701126 25192 raft_consensus.cc:504] T 00000000000000000000000000000000 P 68b74fef56d747d6a6f907ef1c9f83dd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68b74fef56d747d6a6f907ef1c9f83dd" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 36719 } }
I20260925 15:15:53.701609 25192 leader_election.cc:302] T 00000000000000000000000000000000 P 68b74fef56d747d6a6f907ef1c9f83dd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 68b74fef56d747d6a6f907ef1c9f83dd; no voters:
I20260925 15:15:53.702243 25192 leader_election.cc:288] T 00000000000000000000000000000000 P 68b74fef56d747d6a6f907ef1c9f83dd [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 15:15:53.702382 25196 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 68b74fef56d747d6a6f907ef1c9f83dd [term 2 FOLLOWER]: Leader election won for term 2
I20260925 15:15:53.703436 25196 raft_consensus.cc:686] T 00000000000000000000000000000000 P 68b74fef56d747d6a6f907ef1c9f83dd [term 2 LEADER]: Becoming Leader. State: Replica: 68b74fef56d747d6a6f907ef1c9f83dd, State: Running, Role: LEADER
I20260925 15:15:53.704051 25196 consensus_queue.cc:237] T 00000000000000000000000000000000 P 68b74fef56d747d6a6f907ef1c9f83dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68b74fef56d747d6a6f907ef1c9f83dd" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 36719 } }
I20260925 15:15:53.704423 25192 sys_catalog.cc:564] T 00000000000000000000000000000000 P 68b74fef56d747d6a6f907ef1c9f83dd [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:15:53.710747 25197 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68b74fef56d747d6a6f907ef1c9f83dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "68b74fef56d747d6a6f907ef1c9f83dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68b74fef56d747d6a6f907ef1c9f83dd" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 36719 } } }
I20260925 15:15:53.712006 25197 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68b74fef56d747d6a6f907ef1c9f83dd [sys.catalog]: This master's current role is: LEADER
I20260925 15:15:53.712550 25198 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68b74fef56d747d6a6f907ef1c9f83dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68b74fef56d747d6a6f907ef1c9f83dd. Latest consensus state: current_term: 2 leader_uuid: "68b74fef56d747d6a6f907ef1c9f83dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68b74fef56d747d6a6f907ef1c9f83dd" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 36719 } } }
I20260925 15:15:53.713136 25198 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68b74fef56d747d6a6f907ef1c9f83dd [sys.catalog]: This master's current role is: LEADER
I20260925 15:15:53.720202 25205 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:15:53.726205 25205 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:15:53.726971 25192 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 15:15:53.727597 25205 catalog_manager.cc:1259] Loaded cluster ID: eaff2b2162c74873b638de4828f87f77
I20260925 15:15:53.727684 25205 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:15:53.732450 25205 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:15:53.733906 25205 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 68b74fef56d747d6a6f907ef1c9f83dd: Loaded TSK: 0
I20260925 15:15:53.735649 25205 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 15:15:54.090931 25194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:15:54.091408 25194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:15:54.091604 25194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:15:54.103220 25194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:15:54.103408 25194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:15:54.103623 25194 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.1
I20260925 15:15:54.119560 25194 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:44633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.12.8.20:47783/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.1.keytab
--principal=kudu/127.12.8.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.1:36817
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.8.1
--webserver_port=42413
--tserver_master_addrs=127.12.8.62:36719
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.1
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:15:54.121348 25194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:15:54.123057 25194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:15:54.161579 25231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:54.166558 25230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:54.170576 25234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:15:54.171654 25194 server_base.cc:1034] running on GCE node
I20260925 15:15:54.172674 25194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:15:54.176498 25194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:15:54.177807 25194 hybrid_clock.cc:648] HybridClock initialized: now 1790349354177730 us; error 80 us; skew 500 ppm
I20260925 15:15:54.178229 25194 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:15:54 dist-test-slave-p468 krb5kdc[24146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349354, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:15:54.182518 25194 init.cc:377] Logged in from keytab as kudu/127.12.8.1@KRBTEST.COM (short username kudu)
I20260925 15:15:54.185060 25194 webserver.cc:460] Webserver started at http://127.12.8.1:42413/ using document root <none> and password file <none>
I20260925 15:15:54.186111 25194 fs_manager.cc:359] Metadata directory not provided
I20260925 15:15:54.186345 25194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 15:15:54 dist-test-slave-p468 krb5kdc[24146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349354, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.1@KRBTEST.COM for HTTP/127.12.8.20@KRBTEST.COM
I20260925 15:15:54.222724 25194 fs_manager.cc:709] Time spent opening directory manager: real 0.013s user 0.004s sys 0.008s
I20260925 15:15:54.233055 25241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:54.235219 25194 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.006s sys 0.003s
I20260925 15:15:54.235512 25194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ts-0/wal
uuid: "fe814b5e4d664d1793278bd20185f560"
format_stamp: "Formatted at 2026-09-25 15:15:51 on dist-test-slave-p468"
server_key: "2128f202321e060d2a03c3b98cbf10d1"
server_key_iv: "1053c103414276eeaca0a77e9dbaf87e"
server_key_version: "kuduclusterkey@0"
I20260925 15:15:54.237205 25194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:15:54.307507 25194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:15:54.313222 25194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:15:54.313596 25194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:15:54.338258 25194 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:15:54.344513 25194 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:15:54.344733 25194 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:54.344962 25194 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:15:54.345144 25194 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:54.402460 25194 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.1:36817
I20260925 15:15:54.402477 25354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.1:36817 every 8 connection(s)
I20260925 15:15:54.404615 25194 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ts-0/data/info.pb
I20260925 15:15:54.410997 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 25194
I20260925 15:15:54.411881 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.8.20:47783/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.2:43965
--local_ip_for_outbound_sockets=127.12.8.2
--tserver_master_addrs=127.12.8.62:36719
--webserver_port=44265
--webserver_interface=127.12.8.2
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.2.keytab
--principal=kudu/127.12.8.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:44633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.12.8.20:47783/kms with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 15:15:54 dist-test-slave-p468 krb5kdc[24146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349354, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.1@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
I20260925 15:15:54.465644 25358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:54.426705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:36719 (local address 127.12.8.1:56249)
0925 15:15:54.433120 (+ 6415us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:54.433184 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:54.436292 (+ 3108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:54.442518 (+ 6226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:54.442531 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:54.443283 (+ 752us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:15:54.444443 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:54.444473 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:54.447579 (+ 3106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:54.447585 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:54.449922 (+ 2337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:54.449952 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:54.451929 (+ 1977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:54.451937 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:54.452232 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:54.453005 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:15:54.453049 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:15:54.455897 (+ 2848us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:15:54.460819 (+ 4922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:54.460827 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:54.460841 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:54.461402 (+ 561us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:54.461993 (+ 591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:54.461999 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:54.462003 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:54.462261 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:54.463265 (+ 1004us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:15:54.463273 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:15:54.463668 (+ 395us) client_negotiation.cc:769] Sending connection context
0925 15:15:54.463945 (+ 277us) client_negotiation.cc:240] Negotiation successful
0925 15:15:54.464450 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6085,"thread_start_us":5833,"threads_started":1}
I20260925 15:15:54.467856 25357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:54.408574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.1:56249 (local address 127.12.8.62:36719)
0925 15:15:54.429309 (+ 20735us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:54.429333 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:54.433342 (+ 4009us) server_negotiation.cc:407] Connection header received
0925 15:15:54.437092 (+ 3750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:54.437115 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:54.437686 (+ 571us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:54.438152 (+ 466us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:54.444764 (+ 6612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:54.447324 (+ 2560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:54.450129 (+ 2805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:54.451297 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:54.451453 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:54.456189 (+ 4736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:54.456201 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:54.456223 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:54.456309 (+ 86us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:54.460595 (+ 4286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:54.461590 (+ 995us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:54.461602 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:54.461611 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:54.461847 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:54.462402 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:54.462408 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:54.462412 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:54.463103 (+ 691us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:54.463312 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:54.466304 (+ 2992us) server_negotiation.cc:299] Negotiation successful
0925 15:15:54.466703 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20366,"thread_start_us":23583,"threads_started":1}
I20260925 15:15:54.471108 25355 heartbeater.cc:347] Connected to a master server at 127.12.8.62:36719
I20260925 15:15:54.471428 25355 heartbeater.cc:464] Registering TS with master...
I20260925 15:15:54.472285 25355 heartbeater.cc:511] Master 127.12.8.62:36719 requested a full tablet report, sending...
W20260925 15:15:54.475425 25355 heartbeater.cc:644] Failed to heartbeat to 127.12.8.62:36719 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 15:15:54.475632 25355 heartbeater.cc:464] Registering TS with master...
I20260925 15:15:54.475898 25355 heartbeater.cc:511] Master 127.12.8.62:36719 requested a full tablet report, sending...
I20260925 15:15:54.477142 25355 heartbeater.cc:464] Registering TS with master...
I20260925 15:15:54.477433 25355 heartbeater.cc:511] Master 127.12.8.62:36719 requested a full tablet report, sending...
W20260925 15:15:54.478703 25355 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 15:15:54.828853 25359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:15:54.829285 25359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:15:54.829471 25359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:15:54.840731 25359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:15:54.840884 25359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:15:54.841140 25359 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.2
I20260925 15:15:54.853060 25359 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:44633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.12.8.20:47783/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.2.keytab
--principal=kudu/127.12.8.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.2:43965
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.8.2
--webserver_port=44265
--tserver_master_addrs=127.12.8.62:36719
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.2
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:15:54.854421 25359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:15:54.855741 25359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:15:54.872778 25373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:54.872921 25374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:54.877948 25376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:15:54.879165 25359 server_base.cc:1034] running on GCE node
I20260925 15:15:54.879860 25359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:15:54.880877 25359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:15:54.882133 25359 hybrid_clock.cc:648] HybridClock initialized: now 1790349354882074 us; error 52 us; skew 500 ppm
I20260925 15:15:54.882498 25359 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:15:54 dist-test-slave-p468 krb5kdc[24146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349354, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:15:54.885608 25359 init.cc:377] Logged in from keytab as kudu/127.12.8.2@KRBTEST.COM (short username kudu)
I20260925 15:15:54.887328 25359 webserver.cc:460] Webserver started at http://127.12.8.2:44265/ using document root <none> and password file <none>
I20260925 15:15:54.888022 25359 fs_manager.cc:359] Metadata directory not provided
I20260925 15:15:54.888154 25359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 15:15:54 dist-test-slave-p468 krb5kdc[24146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349354, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.2@KRBTEST.COM for HTTP/127.12.8.20@KRBTEST.COM
I20260925 15:15:54.915875 25359 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.007s sys 0.000s
I20260925 15:15:54.934906 25384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:54.936882 25359 fs_manager.cc:725] Time spent opening block manager: real 0.020s user 0.010s sys 0.004s
I20260925 15:15:54.937146 25359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ts-1/wal
uuid: "282ae5bf6e4142d5a298b28ca63b32b0"
format_stamp: "Formatted at 2026-09-25 15:15:51 on dist-test-slave-p468"
server_key: "38fe1e0da2d3bd68a8337c696a6a5105"
server_key_iv: "bccd8cc886b03db2fe9fb4c046d6764d"
server_key_version: "kuduclusterkey@0"
I20260925 15:15:54.938706 25359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:15:54.964403 25359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:15:54.969666 25359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:15:54.970046 25359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:15:54.971339 25359 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:15:54.973109 25359 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:15:54.973213 25359 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:54.973363 25359 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:15:54.973450 25359 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:55.022264 25359 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.2:43965
I20260925 15:15:55.022326 25498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.2:43965 every 8 connection(s)
I20260925 15:15:55.024204 25359 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ts-1/data/info.pb
I20260925 15:15:55.031693 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 25359
I20260925 15:15:55.032541 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.8.20:47783/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.3:36233
--local_ip_for_outbound_sockets=127.12.8.3
--tserver_master_addrs=127.12.8.62:36719
--webserver_port=37547
--webserver_interface=127.12.8.3
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.3.keytab
--principal=kudu/127.12.8.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:44633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.12.8.20:47783/kms with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 15:15:55 dist-test-slave-p468 krb5kdc[24146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349354, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.2@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
Sep 25 15:15:55 dist-test-slave-p468 krb5kdc[24146](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790349355, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:15:55.065927 25501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:55.027269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.2:50651 (local address 127.12.8.62:36719)
0925 15:15:55.030534 (+ 3265us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:55.030538 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:55.030564 (+ 26us) server_negotiation.cc:407] Connection header received
0925 15:15:55.033681 (+ 3117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:55.033690 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:55.033806 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:55.033914 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:55.036138 (+ 2224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:55.037418 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:55.040030 (+ 2612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:55.040729 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:55.040834 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:55.046928 (+ 6094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:55.046938 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:55.046946 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:55.046991 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:55.052792 (+ 5801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:55.063189 (+ 10397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:55.063196 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:55.063200 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:55.063370 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:55.063899 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:55.063903 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:55.063907 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:55.064255 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:55.064394 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:55.065311 (+ 917us) server_negotiation.cc:299] Negotiation successful
0925 15:15:55.065543 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3156,"thread_start_us":3061,"threads_started":1}
I20260925 15:15:55.068408 25502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:55.027815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:36719 (local address 127.12.8.2:50651)
0925 15:15:55.029841 (+ 2026us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:55.029900 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:55.032606 (+ 2706us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:55.034385 (+ 1779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:55.034399 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:55.035048 (+ 649us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:15:55.035924 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:55.035953 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:55.037632 (+ 1679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:55.037640 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:55.039865 (+ 2225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:55.039882 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:55.041177 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:55.041186 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:55.041503 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:55.042362 (+ 859us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:15:55.042405 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:15:55.046660 (+ 4255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:15:55.062296 (+ 15636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:55.062307 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:55.062333 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:55.063017 (+ 684us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:55.063518 (+ 501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:55.063523 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:55.063527 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:55.063774 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:55.064486 (+ 712us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:15:55.064495 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:15:55.064906 (+ 411us) client_negotiation.cc:769] Sending connection context
0925 15:15:55.066493 (+ 1587us) client_negotiation.cc:240] Negotiation successful
0925 15:15:55.067068 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1636,"thread_start_us":6008,"threads_started":1}
I20260925 15:15:55.078408 25499 heartbeater.cc:347] Connected to a master server at 127.12.8.62:36719
I20260925 15:15:55.078759 25499 heartbeater.cc:464] Registering TS with master...
I20260925 15:15:55.079612 25499 heartbeater.cc:511] Master 127.12.8.62:36719 requested a full tablet report, sending...
I20260925 15:15:55.081650 25157 ts_manager.cc:194] Registered new tserver with Master: 282ae5bf6e4142d5a298b28ca63b32b0 (127.12.8.2:43965)
I20260925 15:15:55.083766 25157 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.2@KRBTEST.COM'} at 127.12.8.2:50651
W20260925 15:15:55.409932 25503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:15:55.410406 25503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:15:55.410629 25503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:15:55.418054 25503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:15:55.418205 25503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:15:55.418424 25503 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.3
I20260925 15:15:55.430675 25503 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:44633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.12.8.20:47783/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.3.keytab
--principal=kudu/127.12.8.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.3:36233
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.8.3
--webserver_port=37547
--tserver_master_addrs=127.12.8.62:36719
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.3
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:15:55.432654 25503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:15:55.434521 25503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:15:55.451643 25509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:55.452266 25510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:15:55.460779 25503 server_base.cc:1034] running on GCE node
W20260925 15:15:55.461210 25512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:15:55.462014 25503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:15:55.463297 25503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:15:55.464558 25503 hybrid_clock.cc:648] HybridClock initialized: now 1790349355464501 us; error 46 us; skew 500 ppm
I20260925 15:15:55.464962 25503 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:15:55 dist-test-slave-p468 krb5kdc[24146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349355, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:15:55.473869 25503 init.cc:377] Logged in from keytab as kudu/127.12.8.3@KRBTEST.COM (short username kudu)
I20260925 15:15:55.478817 25355 heartbeater.cc:464] Registering TS with master...
I20260925 15:15:55.479206 25355 heartbeater.cc:511] Master 127.12.8.62:36719 requested a full tablet report, sending...
I20260925 15:15:55.480682 25157 ts_manager.cc:194] Registered new tserver with Master: fe814b5e4d664d1793278bd20185f560 (127.12.8.1:36817)
I20260925 15:15:55.482214 25157 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.1@KRBTEST.COM'} at 127.12.8.1:56249
I20260925 15:15:55.486555 25503 webserver.cc:460] Webserver started at http://127.12.8.3:37547/ using document root <none> and password file <none>
I20260925 15:15:55.487583 25503 fs_manager.cc:359] Metadata directory not provided
I20260925 15:15:55.487749 25503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 15:15:55 dist-test-slave-p468 krb5kdc[24146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349355, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.3@KRBTEST.COM for HTTP/127.12.8.20@KRBTEST.COM
I20260925 15:15:55.515704 25503 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.005s sys 0.000s
Sep 25 15:15:55 dist-test-slave-p468 krb5kdc[24146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349355, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.62@KRBTEST.COM for HTTP/127.12.8.20@KRBTEST.COM
I20260925 15:15:55.521893 25520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:55.523619 25503 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.005s sys 0.001s
I20260925 15:15:55.523815 25503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ts-2/wal
uuid: "e6072e984cd04509a1b66632b5638eed"
format_stamp: "Formatted at 2026-09-25 15:15:52 on dist-test-slave-p468"
server_key: "3b533adc0f61b27007b3dc8a85d6e01e"
server_key_iv: "d22283fd65ff9c446a952fd314bf7760"
server_key_version: "kuduclusterkey@0"
I20260925 15:15:55.524933 25503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:15:55.610577 25503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:15:55.619472 25503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:15:55.619832 25503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:15:55.629031 25503 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:15:55.630894 25503 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:15:55.631009 25503 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:55.631155 25503 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:15:55.631232 25503 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:55.687470 25503 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.3:36233
I20260925 15:15:55.687561 25632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.3:36233 every 8 connection(s)
I20260925 15:15:55.689251 25503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ts-2/data/info.pb
I20260925 15:15:55.696242 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 25503
Sep 25 15:15:55 dist-test-slave-p468 krb5kdc[24146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349355, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.3@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
I20260925 15:15:55.724305 25636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:55.693350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.3:58989 (local address 127.12.8.62:36719)
0925 15:15:55.702766 (+ 9416us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:55.702772 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:55.702792 (+ 20us) server_negotiation.cc:407] Connection header received
0925 15:15:55.702857 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:55.702862 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:55.702968 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:55.703079 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:55.705320 (+ 2241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:55.706556 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:55.708940 (+ 2384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:55.709605 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:55.709706 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:55.716624 (+ 6918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:55.716632 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:55.716636 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:55.716675 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:55.720419 (+ 3744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:55.721387 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:55.721392 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:55.721395 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:55.721546 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:55.722110 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:55.722117 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:55.722121 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:55.722535 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:55.722659 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:55.723671 (+ 1012us) server_negotiation.cc:299] Negotiation successful
0925 15:15:55.723898 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9299,"thread_start_us":9178,"threads_started":1}
I20260925 15:15:55.726975 25635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:55.692685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:36719 (local address 127.12.8.3:58989)
0925 15:15:55.698749 (+ 6064us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:55.698808 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:55.701924 (+ 3116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:55.703550 (+ 1626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:55.703561 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:55.704209 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:15:55.705085 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:55.705112 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:55.706765 (+ 1653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:55.706770 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:55.708769 (+ 1999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:55.708792 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:55.710080 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:55.710089 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:55.710402 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:55.711215 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:15:55.711251 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:15:55.716341 (+ 5090us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:15:55.720614 (+ 4273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:55.720623 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:55.720640 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:55.721236 (+ 596us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:55.721705 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:55.721710 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:55.721713 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:55.721950 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:55.722782 (+ 832us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:15:55.722790 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:15:55.723224 (+ 434us) client_negotiation.cc:769] Sending connection context
0925 15:15:55.724895 (+ 1671us) client_negotiation.cc:240] Negotiation successful
0925 15:15:55.725489 (+ 594us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5755,"thread_start_us":5522,"threads_started":1}
I20260925 15:15:55.729169 25633 heartbeater.cc:347] Connected to a master server at 127.12.8.62:36719
I20260925 15:15:55.729506 25633 heartbeater.cc:464] Registering TS with master...
I20260925 15:15:55.730525 25633 heartbeater.cc:511] Master 127.12.8.62:36719 requested a full tablet report, sending...
I20260925 15:15:55.731998 25637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:55.697241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55742 (local address 127.12.8.62:36719)
0925 15:15:55.714703 (+ 17462us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:55.714709 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:55.714726 (+ 17us) server_negotiation.cc:407] Connection header received
0925 15:15:55.714776 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:55.714781 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:55.714856 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:55.714955 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:55.717110 (+ 2155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:55.718419 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:55.720437 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:55.721153 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:55.721262 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:55.724593 (+ 3331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:55.724600 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:55.724605 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:55.724641 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:55.728132 (+ 3491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:55.728999 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:55.729005 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:55.729008 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:55.729139 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:55.729564 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:55.729568 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:55.729572 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:55.729916 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:55.730162 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:55.731425 (+ 1263us) server_negotiation.cc:299] Negotiation successful
0925 15:15:55.731624 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17328,"thread_start_us":17179,"threads_started":1}
I20260925 15:15:55.732275 25157 ts_manager.cc:194] Registered new tserver with Master: e6072e984cd04509a1b66632b5638eed (127.12.8.3:36233)
I20260925 15:15:55.733385 25157 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.3@KRBTEST.COM'} at 127.12.8.3:58989
I20260925 15:15:55.734726 12320 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:15:55.761085 25637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:55.738054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55758 (local address 127.12.8.62:36719)
0925 15:15:55.738273 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:55.738279 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:55.742799 (+ 4520us) server_negotiation.cc:407] Connection header received
0925 15:15:55.742998 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:55.743004 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:55.743101 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:55.743200 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:55.745242 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:55.746380 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:55.748320 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:55.748917 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:55.749040 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:55.752679 (+ 3639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:55.752686 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:55.752691 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:55.752731 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:55.756545 (+ 3814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:55.757518 (+ 973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:55.757526 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:55.757530 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:55.757729 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:55.758445 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:55.758451 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:55.758455 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:55.758959 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:55.759227 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:55.760463 (+ 1236us) server_negotiation.cc:299] Negotiation successful
0925 15:15:55.760709 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 15:15:55.767331 25156 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55758:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:15:55.778623 25156 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:15:55.819624 25649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:55.806697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.1:36817 (local address 127.0.0.1:49660)
0925 15:15:55.808227 (+ 1530us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:55.808269 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:55.808462 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:55.809817 (+ 1355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:55.809825 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:55.809856 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:15:55.810428 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:55.810455 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:55.813140 (+ 2685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:55.813149 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:55.816173 (+ 3024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:55.816192 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:55.818677 (+ 2485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:55.818684 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:55.818896 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:55.818948 (+ 52us) client_negotiation.cc:769] Sending connection context
0925 15:15:55.819091 (+ 143us) client_negotiation.cc:240] Negotiation successful
0925 15:15:55.819277 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1229,"thread_start_us":1883,"threads_started":1}
I20260925 15:15:55.820698 25650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:55.807492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49660 (local address 127.12.8.1:36817)
0925 15:15:55.809139 (+ 1647us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:55.809148 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:55.809176 (+ 28us) server_negotiation.cc:407] Connection header received
0925 15:15:55.809278 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:55.809287 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:55.809550 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:55.810683 (+ 1133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:15:55.810997 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:55.812937 (+ 1940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:55.816446 (+ 3509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:55.818359 (+ 1913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:55.818569 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:55.818902 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:55.820188 (+ 1286us) server_negotiation.cc:299] Negotiation successful
0925 15:15:55.820369 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1477,"thread_start_us":1295,"threads_started":1}
I20260925 15:15:55.825054 25290 tablet_service.cc:1467] Processing CreateTablet for tablet 83f8d77408f9413a9bf0f8ca13c1d622 (DEFAULT_TABLE table=test-table [id=79e56ab2570f48c186ba571566d1e8a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:15:55.828516 25290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83f8d77408f9413a9bf0f8ca13c1d622. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:15:55.828998 25649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:55.810463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.2:43965 (local address 127.0.0.1:59064)
0925 15:15:55.819891 (+ 9428us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:55.819910 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:55.820022 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:55.821466 (+ 1444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:55.821471 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:55.821498 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:15:55.821694 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:55.821707 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:55.823795 (+ 2088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:55.823801 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:55.825797 (+ 1996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:55.825817 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:55.827893 (+ 2076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:55.827899 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:55.828151 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:55.828187 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 15:15:55.828266 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 15:15:55.828511 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9335,"mutex_wait_us":26,"thread_start_us":13067,"threads_started":1}
I20260925 15:15:55.829222 25652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:55.811048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59064 (local address 127.12.8.2:43965)
0925 15:15:55.819498 (+ 8450us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:55.819511 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:55.820766 (+ 1255us) server_negotiation.cc:407] Connection header received
0925 15:15:55.820880 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:55.820888 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:55.821217 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:55.821471 (+ 254us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:15:55.821921 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:55.823593 (+ 1672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:55.826051 (+ 2458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:55.827686 (+ 1635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:55.828396 (+ 710us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:55.828616 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:55.828684 (+ 68us) server_negotiation.cc:299] Negotiation successful
0925 15:15:55.828872 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8269,"thread_start_us":8118,"threads_started":1}
I20260925 15:15:55.833132 25434 tablet_service.cc:1467] Processing CreateTablet for tablet 83f8d77408f9413a9bf0f8ca13c1d622 (DEFAULT_TABLE table=test-table [id=79e56ab2570f48c186ba571566d1e8a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:15:55.835863 25434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83f8d77408f9413a9bf0f8ca13c1d622. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:15:55.835886 25654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:55.815908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.3:36233 (local address 127.0.0.1:47746)
0925 15:15:55.820441 (+ 4533us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:55.820458 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:55.820585 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:55.822666 (+ 2081us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:55.822671 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:55.822713 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:15:55.823191 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:55.823205 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:55.826471 (+ 3266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:55.826480 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:55.831699 (+ 5219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:55.831716 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:55.834985 (+ 3269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:55.834993 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:55.835183 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:55.835233 (+ 50us) client_negotiation.cc:769] Sending connection context
0925 15:15:55.835329 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 15:15:55.835474 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4397,"mutex_wait_us":1,"thread_start_us":4282,"threads_started":1}
I20260925 15:15:55.837101 25653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:55.814779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47746 (local address 127.12.8.3:36233)
0925 15:15:55.820014 (+ 5235us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:55.820022 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:55.821745 (+ 1723us) server_negotiation.cc:407] Connection header received
0925 15:15:55.821850 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:55.821858 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:55.822100 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:55.822320 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:15:55.823817 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:55.826226 (+ 2409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:55.831888 (+ 5662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:55.834613 (+ 2725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:55.836235 (+ 1622us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:55.836492 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:55.836564 (+ 72us) server_negotiation.cc:299] Negotiation successful
0925 15:15:55.836734 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5056,"thread_start_us":4893,"threads_started":1}
I20260925 15:15:55.840824 25568 tablet_service.cc:1467] Processing CreateTablet for tablet 83f8d77408f9413a9bf0f8ca13c1d622 (DEFAULT_TABLE table=test-table [id=79e56ab2570f48c186ba571566d1e8a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:15:55.842375 25568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83f8d77408f9413a9bf0f8ca13c1d622. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:15:55.858999 25656 tablet_bootstrap.cc:492] T 83f8d77408f9413a9bf0f8ca13c1d622 P fe814b5e4d664d1793278bd20185f560: Bootstrap starting.
I20260925 15:15:55.864495 25656 tablet_bootstrap.cc:654] T 83f8d77408f9413a9bf0f8ca13c1d622 P fe814b5e4d664d1793278bd20185f560: Neither blocks nor log segments found. Creating new log.
I20260925 15:15:55.867744 25656 log.cc:824] T 83f8d77408f9413a9bf0f8ca13c1d622 P fe814b5e4d664d1793278bd20185f560: Log is configured to *not* fsync() on all Append() calls
I20260925 15:15:55.870043 25657 tablet_bootstrap.cc:492] T 83f8d77408f9413a9bf0f8ca13c1d622 P 282ae5bf6e4142d5a298b28ca63b32b0: Bootstrap starting.
I20260925 15:15:55.877192 25657 tablet_bootstrap.cc:654] T 83f8d77408f9413a9bf0f8ca13c1d622 P 282ae5bf6e4142d5a298b28ca63b32b0: Neither blocks nor log segments found. Creating new log.
I20260925 15:15:55.877526 25656 tablet_bootstrap.cc:492] T 83f8d77408f9413a9bf0f8ca13c1d622 P fe814b5e4d664d1793278bd20185f560: No bootstrap required, opened a new log
I20260925 15:15:55.877907 25656 ts_tablet_manager.cc:1397] T 83f8d77408f9413a9bf0f8ca13c1d622 P fe814b5e4d664d1793278bd20185f560: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.005s
I20260925 15:15:55.878801 25657 log.cc:824] T 83f8d77408f9413a9bf0f8ca13c1d622 P 282ae5bf6e4142d5a298b28ca63b32b0: Log is configured to *not* fsync() on all Append() calls
Sep 25 15:15:55 dist-test-slave-p468 krb5kdc[24146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349355, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.62@KRBTEST.COM for HTTP/127.12.8.20@KRBTEST.COM
I20260925 15:15:55.884478 25658 tablet_bootstrap.cc:492] T 83f8d77408f9413a9bf0f8ca13c1d622 P e6072e984cd04509a1b66632b5638eed: Bootstrap starting.
I20260925 15:15:55.889457 25657 tablet_bootstrap.cc:492] T 83f8d77408f9413a9bf0f8ca13c1d622 P 282ae5bf6e4142d5a298b28ca63b32b0: No bootstrap required, opened a new log
I20260925 15:15:55.889846 25657 ts_tablet_manager.cc:1397] T 83f8d77408f9413a9bf0f8ca13c1d622 P 282ae5bf6e4142d5a298b28ca63b32b0: Time spent bootstrapping tablet: real 0.020s user 0.015s sys 0.000s
I20260925 15:15:55.894577 25658 tablet_bootstrap.cc:654] T 83f8d77408f9413a9bf0f8ca13c1d622 P e6072e984cd04509a1b66632b5638eed: Neither blocks nor log segments found. Creating new log.
I20260925 15:15:55.895912 25658 log.cc:824] T 83f8d77408f9413a9bf0f8ca13c1d622 P e6072e984cd04509a1b66632b5638eed: Log is configured to *not* fsync() on all Append() calls
I20260925 15:15:55.898291 25656 raft_consensus.cc:348] T 83f8d77408f9413a9bf0f8ca13c1d622 P fe814b5e4d664d1793278bd20185f560 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe814b5e4d664d1793278bd20185f560" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 36817 } } peers { permanent_uuid: "e6072e984cd04509a1b66632b5638eed" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 36233 } } peers { permanent_uuid: "282ae5bf6e4142d5a298b28ca63b32b0" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 43965 } }
I20260925 15:15:55.898939 25656 raft_consensus.cc:374] T 83f8d77408f9413a9bf0f8ca13c1d622 P fe814b5e4d664d1793278bd20185f560 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:15:55.899112 25656 raft_consensus.cc:729] T 83f8d77408f9413a9bf0f8ca13c1d622 P fe814b5e4d664d1793278bd20185f560 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe814b5e4d664d1793278bd20185f560, State: Initialized, Role: FOLLOWER
I20260925 15:15:55.900029 25656 consensus_queue.cc:260] T 83f8d77408f9413a9bf0f8ca13c1d622 P fe814b5e4d664d1793278bd20185f560 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe814b5e4d664d1793278bd20185f560" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 36817 } } peers { permanent_uuid: "e6072e984cd04509a1b66632b5638eed" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 36233 } } peers { permanent_uuid: "282ae5bf6e4142d5a298b28ca63b32b0" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 43965 } }
I20260925 15:15:55.958319 25656 ts_tablet_manager.cc:1428] T 83f8d77408f9413a9bf0f8ca13c1d622 P fe814b5e4d664d1793278bd20185f560: Time spent starting tablet: real 0.080s user 0.047s sys 0.020s
I20260925 15:15:55.958432 25355 heartbeater.cc:503] Master 127.12.8.62:36719 was elected leader, sending a full tablet report...
I20260925 15:15:55.960775 25658 tablet_bootstrap.cc:492] T 83f8d77408f9413a9bf0f8ca13c1d622 P e6072e984cd04509a1b66632b5638eed: No bootstrap required, opened a new log
I20260925 15:15:55.961095 25658 ts_tablet_manager.cc:1397] T 83f8d77408f9413a9bf0f8ca13c1d622 P e6072e984cd04509a1b66632b5638eed: Time spent bootstrapping tablet: real 0.077s user 0.023s sys 0.044s
I20260925 15:15:55.968166 25657 raft_consensus.cc:348] T 83f8d77408f9413a9bf0f8ca13c1d622 P 282ae5bf6e4142d5a298b28ca63b32b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe814b5e4d664d1793278bd20185f560" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 36817 } } peers { permanent_uuid: "e6072e984cd04509a1b66632b5638eed" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 36233 } } peers { permanent_uuid: "282ae5bf6e4142d5a298b28ca63b32b0" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 43965 } }
I20260925 15:15:55.968645 25657 raft_consensus.cc:374] T 83f8d77408f9413a9bf0f8ca13c1d622 P 282ae5bf6e4142d5a298b28ca63b32b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:15:55.970850 25657 raft_consensus.cc:729] T 83f8d77408f9413a9bf0f8ca13c1d622 P 282ae5bf6e4142d5a298b28ca63b32b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 282ae5bf6e4142d5a298b28ca63b32b0, State: Initialized, Role: FOLLOWER
I20260925 15:15:55.971846 25658 raft_consensus.cc:348] T 83f8d77408f9413a9bf0f8ca13c1d622 P e6072e984cd04509a1b66632b5638eed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe814b5e4d664d1793278bd20185f560" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 36817 } } peers { permanent_uuid: "e6072e984cd04509a1b66632b5638eed" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 36233 } } peers { permanent_uuid: "282ae5bf6e4142d5a298b28ca63b32b0" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 43965 } }
I20260925 15:15:55.971869 25657 consensus_queue.cc:260] T 83f8d77408f9413a9bf0f8ca13c1d622 P 282ae5bf6e4142d5a298b28ca63b32b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe814b5e4d664d1793278bd20185f560" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 36817 } } peers { permanent_uuid: "e6072e984cd04509a1b66632b5638eed" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 36233 } } peers { permanent_uuid: "282ae5bf6e4142d5a298b28ca63b32b0" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 43965 } }
I20260925 15:15:55.972230 25658 raft_consensus.cc:374] T 83f8d77408f9413a9bf0f8ca13c1d622 P e6072e984cd04509a1b66632b5638eed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:15:55.972362 25658 raft_consensus.cc:729] T 83f8d77408f9413a9bf0f8ca13c1d622 P e6072e984cd04509a1b66632b5638eed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6072e984cd04509a1b66632b5638eed, State: Initialized, Role: FOLLOWER
I20260925 15:15:55.973001 25658 consensus_queue.cc:260] T 83f8d77408f9413a9bf0f8ca13c1d622 P e6072e984cd04509a1b66632b5638eed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe814b5e4d664d1793278bd20185f560" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 36817 } } peers { permanent_uuid: "e6072e984cd04509a1b66632b5638eed" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 36233 } } peers { permanent_uuid: "282ae5bf6e4142d5a298b28ca63b32b0" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 43965 } }
I20260925 15:15:55.978317 25657 ts_tablet_manager.cc:1428] T 83f8d77408f9413a9bf0f8ca13c1d622 P 282ae5bf6e4142d5a298b28ca63b32b0: Time spent starting tablet: real 0.088s user 0.020s sys 0.001s
I20260925 15:15:55.979137 25499 heartbeater.cc:503] Master 127.12.8.62:36719 was elected leader, sending a full tablet report...
I20260925 15:15:55.986712 25633 heartbeater.cc:503] Master 127.12.8.62:36719 was elected leader, sending a full tablet report...
I20260925 15:15:55.987142 25658 ts_tablet_manager.cc:1428] T 83f8d77408f9413a9bf0f8ca13c1d622 P e6072e984cd04509a1b66632b5638eed: Time spent starting tablet: real 0.026s user 0.013s sys 0.013s
W20260925 15:15:56.028746 25500 tablet.cc:2369] T 83f8d77408f9413a9bf0f8ca13c1d622 P 282ae5bf6e4142d5a298b28ca63b32b0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:15:56.183256 25356 tablet.cc:2369] T 83f8d77408f9413a9bf0f8ca13c1d622 P fe814b5e4d664d1793278bd20185f560: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:15:56.197072 25634 tablet.cc:2369] T 83f8d77408f9413a9bf0f8ca13c1d622 P e6072e984cd04509a1b66632b5638eed: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:15:56.270864 25663 raft_consensus.cc:482] T 83f8d77408f9413a9bf0f8ca13c1d622 P 282ae5bf6e4142d5a298b28ca63b32b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:15:56.271170 25663 raft_consensus.cc:504] T 83f8d77408f9413a9bf0f8ca13c1d622 P 282ae5bf6e4142d5a298b28ca63b32b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe814b5e4d664d1793278bd20185f560" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 36817 } } peers { permanent_uuid: "e6072e984cd04509a1b66632b5638eed" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 36233 } } peers { permanent_uuid: "282ae5bf6e4142d5a298b28ca63b32b0" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 43965 } }
I20260925 15:15:56.272821 25663 leader_election.cc:288] T 83f8d77408f9413a9bf0f8ca13c1d622 P 282ae5bf6e4142d5a298b28ca63b32b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe814b5e4d664d1793278bd20185f560 (127.12.8.1:36817), e6072e984cd04509a1b66632b5638eed (127.12.8.3:36233)
I20260925 15:15:56.279683 25650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:56.273374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.2:33019 (local address 127.12.8.1:36817)
0925 15:15:56.273591 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:56.273597 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:56.273767 (+ 170us) server_negotiation.cc:407] Connection header received
0925 15:15:56.274020 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:56.274026 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:56.274122 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:56.274189 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:15:56.275138 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:56.276146 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:56.277812 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:56.278648 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:56.278807 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:56.278928 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:56.279304 (+ 376us) server_negotiation.cc:299] Negotiation successful
0925 15:15:56.279430 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 15:15:56.279814 25672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:56.273077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.1:36817 (local address 127.12.8.2:33019)
0925 15:15:56.273670 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:56.273685 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:56.273845 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:56.274494 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:56.274498 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:56.274531 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:15:56.274939 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:56.274953 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:56.276278 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:56.276285 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:56.277654 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:56.277665 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:56.278885 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:56.278891 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:56.279144 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:56.279172 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 15:15:56.279279 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 15:15:56.279415 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"mutex_wait_us":45,"thread_start_us":291,"threads_started":1}
I20260925 15:15:56.280869 25310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83f8d77408f9413a9bf0f8ca13c1d622" candidate_uuid: "282ae5bf6e4142d5a298b28ca63b32b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe814b5e4d664d1793278bd20185f560" is_pre_election: true
I20260925 15:15:56.281378 25310 raft_consensus.cc:2454] T 83f8d77408f9413a9bf0f8ca13c1d622 P fe814b5e4d664d1793278bd20185f560 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 282ae5bf6e4142d5a298b28ca63b32b0 in term 0.
I20260925 15:15:56.281670 25653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:56.274033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.2:33189 (local address 127.12.8.3:36233)
0925 15:15:56.274248 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:56.274254 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:56.274861 (+ 607us) server_negotiation.cc:407] Connection header received
0925 15:15:56.275041 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:56.275047 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:56.275116 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:56.275205 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:15:56.275834 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:56.276940 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:56.279457 (+ 2517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:56.280609 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:56.280726 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:56.280819 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:56.281311 (+ 492us) server_negotiation.cc:299] Negotiation successful
0925 15:15:56.281445 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 15:15:56.282263 25389 leader_election.cc:302] T 83f8d77408f9413a9bf0f8ca13c1d622 P 282ae5bf6e4142d5a298b28ca63b32b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 282ae5bf6e4142d5a298b28ca63b32b0, fe814b5e4d664d1793278bd20185f560; no voters:
I20260925 15:15:56.282388 25673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:56.273739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.3:36233 (local address 127.12.8.2:33189)
0925 15:15:56.274739 (+ 1000us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:56.274756 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:56.274903 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:56.275237 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:56.275242 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:56.275271 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:15:56.275666 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:56.275677 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:56.277130 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:56.277136 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:56.279279 (+ 2143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:56.279294 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:56.280967 (+ 1673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:56.280973 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:56.281132 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:56.281163 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 15:15:56.281841 (+ 678us) client_negotiation.cc:240] Negotiation successful
0925 15:15:56.281981 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":851,"mutex_wait_us":69,"thread_start_us":648,"threads_started":1}
I20260925 15:15:56.282684 25663 raft_consensus.cc:2781] T 83f8d77408f9413a9bf0f8ca13c1d622 P 282ae5bf6e4142d5a298b28ca63b32b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:15:56.282851 25663 raft_consensus.cc:482] T 83f8d77408f9413a9bf0f8ca13c1d622 P 282ae5bf6e4142d5a298b28ca63b32b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:15:56.282979 25663 raft_consensus.cc:3037] T 83f8d77408f9413a9bf0f8ca13c1d622 P 282ae5bf6e4142d5a298b28ca63b32b0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:15:56.283483 25588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83f8d77408f9413a9bf0f8ca13c1d622" candidate_uuid: "282ae5bf6e4142d5a298b28ca63b32b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6072e984cd04509a1b66632b5638eed" is_pre_election: true
I20260925 15:15:56.283938 25588 raft_consensus.cc:2454] T 83f8d77408f9413a9bf0f8ca13c1d622 P e6072e984cd04509a1b66632b5638eed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 282ae5bf6e4142d5a298b28ca63b32b0 in term 0.
I20260925 15:15:56.285670 25663 raft_consensus.cc:504] T 83f8d77408f9413a9bf0f8ca13c1d622 P 282ae5bf6e4142d5a298b28ca63b32b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe814b5e4d664d1793278bd20185f560" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 36817 } } peers { permanent_uuid: "e6072e984cd04509a1b66632b5638eed" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 36233 } } peers { permanent_uuid: "282ae5bf6e4142d5a298b28ca63b32b0" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 43965 } }
I20260925 15:15:56.286314 25663 leader_election.cc:288] T 83f8d77408f9413a9bf0f8ca13c1d622 P 282ae5bf6e4142d5a298b28ca63b32b0 [CANDIDATE]: Term 1 election: Requested vote from peers fe814b5e4d664d1793278bd20185f560 (127.12.8.1:36817), e6072e984cd04509a1b66632b5638eed (127.12.8.3:36233)
I20260925 15:15:56.286746 25310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83f8d77408f9413a9bf0f8ca13c1d622" candidate_uuid: "282ae5bf6e4142d5a298b28ca63b32b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe814b5e4d664d1793278bd20185f560"
I20260925 15:15:56.286841 25588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83f8d77408f9413a9bf0f8ca13c1d622" candidate_uuid: "282ae5bf6e4142d5a298b28ca63b32b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6072e984cd04509a1b66632b5638eed"
I20260925 15:15:56.286962 25310 raft_consensus.cc:3037] T 83f8d77408f9413a9bf0f8ca13c1d622 P fe814b5e4d664d1793278bd20185f560 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:15:56.287050 25588 raft_consensus.cc:3037] T 83f8d77408f9413a9bf0f8ca13c1d622 P e6072e984cd04509a1b66632b5638eed [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:15:56.289198 25310 raft_consensus.cc:2454] T 83f8d77408f9413a9bf0f8ca13c1d622 P fe814b5e4d664d1793278bd20185f560 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 282ae5bf6e4142d5a298b28ca63b32b0 in term 1.
I20260925 15:15:56.289548 25389 leader_election.cc:302] T 83f8d77408f9413a9bf0f8ca13c1d622 P 282ae5bf6e4142d5a298b28ca63b32b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 282ae5bf6e4142d5a298b28ca63b32b0, fe814b5e4d664d1793278bd20185f560; no voters:
I20260925 15:15:56.289820 25663 raft_consensus.cc:2781] T 83f8d77408f9413a9bf0f8ca13c1d622 P 282ae5bf6e4142d5a298b28ca63b32b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:15:56.289991 25588 raft_consensus.cc:2454] T 83f8d77408f9413a9bf0f8ca13c1d622 P e6072e984cd04509a1b66632b5638eed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 282ae5bf6e4142d5a298b28ca63b32b0 in term 1.
I20260925 15:15:56.290612 25663 raft_consensus.cc:686] T 83f8d77408f9413a9bf0f8ca13c1d622 P 282ae5bf6e4142d5a298b28ca63b32b0 [term 1 LEADER]: Becoming Leader. State: Replica: 282ae5bf6e4142d5a298b28ca63b32b0, State: Running, Role: LEADER
I20260925 15:15:56.291074 25663 consensus_queue.cc:237] T 83f8d77408f9413a9bf0f8ca13c1d622 P 282ae5bf6e4142d5a298b28ca63b32b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe814b5e4d664d1793278bd20185f560" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 36817 } } peers { permanent_uuid: "e6072e984cd04509a1b66632b5638eed" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 36233 } } peers { permanent_uuid: "282ae5bf6e4142d5a298b28ca63b32b0" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 43965 } }
I20260925 15:15:56.295817 25156 catalog_manager.cc:5495] T 83f8d77408f9413a9bf0f8ca13c1d622 P 282ae5bf6e4142d5a298b28ca63b32b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 282ae5bf6e4142d5a298b28ca63b32b0 (127.12.8.2). New cstate: current_term: 1 leader_uuid: "282ae5bf6e4142d5a298b28ca63b32b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe814b5e4d664d1793278bd20185f560" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 36817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6072e984cd04509a1b66632b5638eed" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 36233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "282ae5bf6e4142d5a298b28ca63b32b0" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 43965 } health_report { overall_health: HEALTHY } } }
I20260925 15:15:56.389916 25679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:56.384104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59078 (local address 127.12.8.2:43965)
0925 15:15:56.384758 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:56.384762 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:56.384779 (+ 17us) server_negotiation.cc:407] Connection header received
0925 15:15:56.384994 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:56.384998 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:56.385088 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:56.385236 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:15:56.385584 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:56.386327 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:56.387775 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:56.388314 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:56.388413 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:56.388811 (+ 398us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:15:56.389323 (+ 512us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:15:56.389434 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:56.389513 (+ 79us) server_negotiation.cc:299] Negotiation successful
0925 15:15:56.389571 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":438,"threads_started":1}
I20260925 15:15:56.397955 25588 raft_consensus.cc:1261] T 83f8d77408f9413a9bf0f8ca13c1d622 P e6072e984cd04509a1b66632b5638eed [term 1 FOLLOWER]: Refusing update from remote peer 282ae5bf6e4142d5a298b28ca63b32b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:15:56.398008 25310 raft_consensus.cc:1261] T 83f8d77408f9413a9bf0f8ca13c1d622 P fe814b5e4d664d1793278bd20185f560 [term 1 FOLLOWER]: Refusing update from remote peer 282ae5bf6e4142d5a298b28ca63b32b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:15:56.398959 25663 consensus_queue.cc:1035] T 83f8d77408f9413a9bf0f8ca13c1d622 P 282ae5bf6e4142d5a298b28ca63b32b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6072e984cd04509a1b66632b5638eed" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 36233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:15:56.399345 25674 consensus_queue.cc:1035] T 83f8d77408f9413a9bf0f8ca13c1d622 P 282ae5bf6e4142d5a298b28ca63b32b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe814b5e4d664d1793278bd20185f560" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 36817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:15:56.412401 25680 mvcc.cc:205] Tried to move back new op lower bound from 7333270963795939328 to 7333270963372003328. Current Snapshot: MvccSnapshot[applied={T|T < 7333270963795939328}]
I20260925 15:15:56.425619 25156 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55758:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:15:56.425908 25156 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55758:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:15:56.430914 25156 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 68b74fef56d747d6a6f907ef1c9f83dd: Sending DeleteTablet for 3 replicas of tablet 83f8d77408f9413a9bf0f8ca13c1d622
I20260925 15:15:56.432096 25290 tablet_service.cc:1514] Processing DeleteTablet for tablet 83f8d77408f9413a9bf0f8ca13c1d622 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:15:56 UTC) from {username='kudu', principal='kudu/127.12.8.62@KRBTEST.COM'} at 127.0.0.1:49660
I20260925 15:15:56.432188 25568 tablet_service.cc:1514] Processing DeleteTablet for tablet 83f8d77408f9413a9bf0f8ca13c1d622 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:15:56 UTC) from {username='kudu', principal='kudu/127.12.8.62@KRBTEST.COM'} at 127.0.0.1:47746
I20260925 15:15:56.432385 25434 tablet_service.cc:1514] Processing DeleteTablet for tablet 83f8d77408f9413a9bf0f8ca13c1d622 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:15:56 UTC) from {username='kudu', principal='kudu/127.12.8.62@KRBTEST.COM'} at 127.0.0.1:59064
I20260925 15:15:56.438886 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 25194
I20260925 15:15:56.446805 25691 tablet_replica.cc:330] T 83f8d77408f9413a9bf0f8ca13c1d622 P e6072e984cd04509a1b66632b5638eed: stopping tablet replica
I20260925 15:15:56.447180 25690 tablet_replica.cc:330] T 83f8d77408f9413a9bf0f8ca13c1d622 P fe814b5e4d664d1793278bd20185f560: stopping tablet replica
I20260925 15:15:56.447314 25691 raft_consensus.cc:2229] T 83f8d77408f9413a9bf0f8ca13c1d622 P e6072e984cd04509a1b66632b5638eed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:15:56.447561 25690 raft_consensus.cc:2229] T 83f8d77408f9413a9bf0f8ca13c1d622 P fe814b5e4d664d1793278bd20185f560 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:15:56.447822 25691 raft_consensus.cc:2258] T 83f8d77408f9413a9bf0f8ca13c1d622 P e6072e984cd04509a1b66632b5638eed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:15:56.448104 25690 raft_consensus.cc:2258] T 83f8d77408f9413a9bf0f8ca13c1d622 P fe814b5e4d664d1793278bd20185f560 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:15:56.448535 25692 tablet_replica.cc:330] T 83f8d77408f9413a9bf0f8ca13c1d622 P 282ae5bf6e4142d5a298b28ca63b32b0: stopping tablet replica
I20260925 15:15:56.448892 25692 raft_consensus.cc:2229] T 83f8d77408f9413a9bf0f8ca13c1d622 P 282ae5bf6e4142d5a298b28ca63b32b0 [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:15:56.449482 25692 raft_consensus.cc:2258] T 83f8d77408f9413a9bf0f8ca13c1d622 P 282ae5bf6e4142d5a298b28ca63b32b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:15:56.450098 25691 ts_tablet_manager.cc:1905] T 83f8d77408f9413a9bf0f8ca13c1d622 P e6072e984cd04509a1b66632b5638eed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:15:56.450691 25690 ts_tablet_manager.cc:1905] T 83f8d77408f9413a9bf0f8ca13c1d622 P fe814b5e4d664d1793278bd20185f560: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:15:56.451287 25692 ts_tablet_manager.cc:1905] T 83f8d77408f9413a9bf0f8ca13c1d622 P 282ae5bf6e4142d5a298b28ca63b32b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:15:56.457202 25691 ts_tablet_manager.cc:1918] T 83f8d77408f9413a9bf0f8ca13c1d622 P e6072e984cd04509a1b66632b5638eed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:15:56.457417 25691 log.cc:1196] T 83f8d77408f9413a9bf0f8ca13c1d622 P e6072e984cd04509a1b66632b5638eed: Deleting WAL directory at /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ts-2/wal/wals/83f8d77408f9413a9bf0f8ca13c1d622
I20260925 15:15:56.457504 25690 ts_tablet_manager.cc:1918] T 83f8d77408f9413a9bf0f8ca13c1d622 P fe814b5e4d664d1793278bd20185f560: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:15:56.457693 25690 log.cc:1196] T 83f8d77408f9413a9bf0f8ca13c1d622 P fe814b5e4d664d1793278bd20185f560: Deleting WAL directory at /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ts-0/wal/wals/83f8d77408f9413a9bf0f8ca13c1d622
I20260925 15:15:56.458072 25691 ts_tablet_manager.cc:1939] T 83f8d77408f9413a9bf0f8ca13c1d622 P e6072e984cd04509a1b66632b5638eed: Deleting consensus metadata
I20260925 15:15:56.458283 25690 ts_tablet_manager.cc:1939] T 83f8d77408f9413a9bf0f8ca13c1d622 P fe814b5e4d664d1793278bd20185f560: Deleting consensus metadata
I20260925 15:15:56.460036 25143 catalog_manager.cc:4841] TS fe814b5e4d664d1793278bd20185f560 (127.12.8.1:36817): tablet 83f8d77408f9413a9bf0f8ca13c1d622 (table test-table [id=79e56ab2570f48c186ba571566d1e8a2]) successfully deleted
I20260925 15:15:56.460315 25143 catalog_manager.cc:4841] TS e6072e984cd04509a1b66632b5638eed (127.12.8.3:36233): tablet 83f8d77408f9413a9bf0f8ca13c1d622 (table test-table [id=79e56ab2570f48c186ba571566d1e8a2]) successfully deleted
I20260925 15:15:56.462740 25692 ts_tablet_manager.cc:1918] T 83f8d77408f9413a9bf0f8ca13c1d622 P 282ae5bf6e4142d5a298b28ca63b32b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:15:56.462908 25692 log.cc:1196] T 83f8d77408f9413a9bf0f8ca13c1d622 P 282ae5bf6e4142d5a298b28ca63b32b0: Deleting WAL directory at /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349162370577-12320-0/minicluster-data/ts-1/wal/wals/83f8d77408f9413a9bf0f8ca13c1d622
I20260925 15:15:56.463440 25692 ts_tablet_manager.cc:1939] T 83f8d77408f9413a9bf0f8ca13c1d622 P 282ae5bf6e4142d5a298b28ca63b32b0: Deleting consensus metadata
I20260925 15:15:56.465101 25144 catalog_manager.cc:4841] TS 282ae5bf6e4142d5a298b28ca63b32b0 (127.12.8.2:43965): tablet 83f8d77408f9413a9bf0f8ca13c1d622 (table test-table [id=79e56ab2570f48c186ba571566d1e8a2]) successfully deleted
I20260925 15:15:56.474278 25650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:56.439755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49668 (local address 127.12.8.1:36817)
0925 15:15:56.440071 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:56.440081 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:56.460678 (+ 20597us) server_negotiation.cc:407] Connection header received
0925 15:15:56.460886 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:56.460892 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:56.460988 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:56.461082 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:56.462474 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:56.463376 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:56.464655 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:56.465158 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:56.465293 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:56.468507 (+ 3214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:56.468518 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:56.468532 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:56.468565 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:56.471433 (+ 2868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:56.472267 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:56.472274 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:56.472276 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:56.472402 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:56.472821 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:56.472825 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:56.472827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:56.473192 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:56.473424 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:56.473768 (+ 344us) server_negotiation.cc:299] Negotiation successful
0925 15:15:56.473926 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148}
I20260925 15:15:56.475057 25350 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:15:56.565424 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 25194
I20260925 15:15:56.581737 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 25359
I20260925 15:15:56.597079 25679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:56.582589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59082 (local address 127.12.8.2:43965)
0925 15:15:56.582774 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:56.582780 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:56.582803 (+ 23us) server_negotiation.cc:407] Connection header received
0925 15:15:56.582981 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:56.582987 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:56.583070 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:56.583180 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:56.584421 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:56.585662 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:56.586942 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:56.587612 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:56.587738 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:56.590227 (+ 2489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:56.590246 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:56.590267 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:56.590307 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:56.594153 (+ 3846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:56.595010 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:56.595020 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:56.595024 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:56.595181 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:56.595527 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:56.595532 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:56.595536 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:56.595947 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:56.596271 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:56.596418 (+ 147us) server_negotiation.cc:299] Negotiation successful
0925 15:15:56.596688 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 15:15:56.597867 25494 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:15:56.709429 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 25359
I20260925 15:15:56.728296 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 25503
I20260925 15:15:56.741703 25653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:56.729137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47754 (local address 127.12.8.3:36233)
0925 15:15:56.729326 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:56.729331 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:56.729349 (+ 18us) server_negotiation.cc:407] Connection header received
0925 15:15:56.729479 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:56.729483 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:56.729563 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:56.729652 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:56.731222 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:56.732043 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:56.733274 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:56.733663 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:56.733753 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:56.736240 (+ 2487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:56.736253 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:56.736269 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:56.736299 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:56.739194 (+ 2895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:56.739801 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:56.739808 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:56.739810 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:56.739928 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:56.740240 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:56.740243 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:56.740245 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:56.740583 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:56.740797 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:56.741155 (+ 358us) server_negotiation.cc:299] Negotiation successful
0925 15:15:56.741347 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 15:15:56.742446 25628 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:15:56.829487 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 25503
I20260925 15:15:56.845991 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 25124
I20260925 15:15:56.847285 25187 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:15:56.959290 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 25124
I20260925 15:15:56.980007 12320 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 15:15:57.314908 12320 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 15:15:57.315153 12320 mini_ranger.cc:66] Stopping Ranger...
I20260925 15:15:57.660518 12320 mini_ranger.cc:68] Stopped Ranger
2026-09-25 15:15:57.660 UTC [24176] LOG: received smart shutdown request
2026-09-25 15:15:57.664 UTC [24176] LOG: background worker "logical replication launcher" (PID 24185) exited with exit code 1
2026-09-25 15:15:57.664 UTC [24180] LOG: shutting down
2026-09-25 15:15:57.835 UTC [24176] LOG: database system is shut down
2026-09-25T15:15:57Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (52761 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349162370577-12320-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:15:58 dist-test-slave-p468 krb5kdc[25706](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:15:58 dist-test-slave-p468 krb5kdc[25706](info): set up 2 sockets
Sep 25 15:15:58 dist-test-slave-p468 krb5kdc[25706](info): commencing operation
krb5kdc: starting...
W20260925 15:15:59.078591 12320 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.001s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349162370577-12320-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349162370577-12320-0/krb5kdc/test-user.keytab.
Sep 25 15:15:59 dist-test-slave-p468 krb5kdc[25706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:15:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:15:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.8.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab.
Entry for principal kudu/127.12.8.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab.
I20260925 15:15:59.301347 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349162370577-12320-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349162370577-12320-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349162370577-12320-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.62:38915
--webserver_interface=127.12.8.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab
--principal=kudu/127.12.8.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:36529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.62:38915,127.12.8.61:41161,127.12.8.60:33203
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:15:59.622990 25719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:15:59.623319 25719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:15:59.623497 25719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:15:59.630625 25719 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:15:59.630771 25719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:15:59.630877 25719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:15:59.630983 25719 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:15:59.631071 25719 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:15:59.640810 25719 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:36529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349162370577-12320-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349162370577-12320-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.8.62:38915,127.12.8.61:41161,127.12.8.60:33203
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab
--principal=kudu/127.12.8.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.62:38915
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.8.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349162370577-12320-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:15:59.642151 25719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:15:59.643764 25719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:15:59.655771 25727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:59.655913 25724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:15:59.656358 25725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:15:59.656767 25719 server_base.cc:1034] running on GCE node
I20260925 15:15:59.657514 25719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:15:59.658872 25719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:15:59.660104 25719 hybrid_clock.cc:648] HybridClock initialized: now 1790349359660064 us; error 35 us; skew 500 ppm
I20260925 15:15:59.660447 25719 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:15:59 dist-test-slave-p468 krb5kdc[25706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349359, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:15:59.663646 25719 init.cc:377] Logged in from keytab as kudu/127.12.8.62@KRBTEST.COM (short username kudu)
I20260925 15:15:59.665104 25719 webserver.cc:460] Webserver started at http://127.12.8.62:42889/ using document root <none> and password file <none>
I20260925 15:15:59.665760 25719 fs_manager.cc:359] Metadata directory not provided
I20260925 15:15:59.665860 25719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:15:59.666103 25719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:15:59.669132 25719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349162370577-12320-0/minicluster-data/master-0/data/instance:
uuid: "89eec0efbc2a4b58a3e2298936464134"
format_stamp: "Formatted at 2026-09-25 15:15:59 on dist-test-slave-p468"
server_key: "5acec9d63e24243ec983fe7385d4d000"
server_key_iv: "0fe62836564b2020d50fbd50575a435e"
server_key_version: "encryptionkey@0"
I20260925 15:15:59.669899 25719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349162370577-12320-0/minicluster-data/master-0/wal/instance:
uuid: "89eec0efbc2a4b58a3e2298936464134"
format_stamp: "Formatted at 2026-09-25 15:15:59 on dist-test-slave-p468"
server_key: "5acec9d63e24243ec983fe7385d4d000"
server_key_iv: "0fe62836564b2020d50fbd50575a435e"
server_key_version: "encryptionkey@0"
I20260925 15:15:59.674993 25719 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260925 15:15:59.678339 25734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:15:59.679672 25719 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 15:15:59.679808 25719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349162370577-12320-0/minicluster-data/master-0/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349162370577-12320-0/minicluster-data/master-0/wal
uuid: "89eec0efbc2a4b58a3e2298936464134"
format_stamp: "Formatted at 2026-09-25 15:15:59 on dist-test-slave-p468"
server_key: "5acec9d63e24243ec983fe7385d4d000"
server_key_iv: "0fe62836564b2020d50fbd50575a435e"
server_key_version: "encryptionkey@0"
I20260925 15:15:59.679965 25719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349162370577-12320-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349162370577-12320-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349162370577-12320-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:15:59.723842 25719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:15:59.728372 25719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:15:59.728822 25719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:15:59.748600 25719 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.62:38915
I20260925 15:15:59.748641 25785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.62:38915 every 8 connection(s)
I20260925 15:15:59.749961 25719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
I20260925 15:15:59.753815 25786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:15:59.754899 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 25719
I20260925 15:15:59.755146 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349162370577-12320-0/minicluster-data/master-0/wal/instance
I20260925 15:15:59.755836 12320 external_mini_cluster.cc:1428] Setting key 70e4e3fc140e0e14e3a9d459affefa2a
I20260925 15:15:59.760389 25786 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.8.62" port: 38915 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:15:59 dist-test-slave-p468 krb5kdc[25706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349359, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.62@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.8.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.61@KRBTEST.COM" created.
I20260925 15:15:59.783529 25790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:59.763508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:38915 (local address 127.0.0.1:50972)
0925 15:15:59.764644 (+ 1136us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:59.764676 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 15:15:59.766464 (+ 1788us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:15:59.767285 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:15:59.767292 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:15:59.768025 (+ 733us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:15:59.768366 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:59.768384 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:59.769928 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:59.769935 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:59.771082 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:15:59.771094 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:15:59.772022 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:15:59.772028 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:15:59.772259 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:59.772939 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:15:59.772961 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:15:59.775390 (+ 2429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:15:59.779040 (+ 3650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:59.779054 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:59.779067 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:59.779493 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:59.779920 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:15:59.779928 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:15:59.779934 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:15:59.780104 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:15:59.780881 (+ 777us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:15:59.780895 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:15:59.781132 (+ 237us) client_negotiation.cc:769] Sending connection context
0925 15:15:59.781399 (+ 267us) client_negotiation.cc:240] Negotiation successful
0925 15:15:59.782051 (+ 652us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":946,"mutex_wait_us":197,"spinlock_wait_cycles":14208,"thread_start_us":732,"threads_started":1}
I20260925 15:15:59.783552 25789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:59.763507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50972 (local address 127.12.8.62:38915)
0925 15:15:59.764341 (+ 834us) server_negotiation.cc:206] Beginning negotiation
0925 15:15:59.764359 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 15:15:59.764810 (+ 451us) server_negotiation.cc:407] Connection header received
0925 15:15:59.766824 (+ 2014us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:15:59.766829 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:15:59.767057 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:15:59.767257 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:15:59.768567 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:59.769762 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:59.771219 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:15:59.771774 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:15:59.771988 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:15:59.775643 (+ 3655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:15:59.775658 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:15:59.775677 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:15:59.775726 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:15:59.778886 (+ 3160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:59.779634 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:59.779650 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:59.779652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:59.779793 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:15:59.780224 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:15:59.780236 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:15:59.780239 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:15:59.780730 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:15:59.780962 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0925 15:15:59.781538 (+ 576us) server_negotiation.cc:299] Negotiation successful
0925 15:15:59.781926 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":60,"server-negotiator.queue_time_us":601,"thread_start_us":399,"threads_started":1}
I20260925 15:15:59.787034 25786 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.8.61" port: 41161 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:15:59.788177 25790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:59.787771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.61:41161 (local address 127.0.0.1:44464)
0925 15:15:59.787947 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:59.788057 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.61:41161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260925 15:15:59.788477 25735 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.8.61:41161: connect: Connection refused (error 111)
I20260925 15:15:59.790027 25790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:59.789684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.61:41161 (local address 127.0.0.1:44480)
0925 15:15:59.789827 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:59.789902 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.61:41161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260925 15:15:59.790264 25786 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.8.61:41161: Network error: Client connection negotiation failed: client connection to 127.12.8.61:41161: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.61.keytab.
Entry for principal kudu/127.12.8.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.61.keytab.
I20260925 15:15:59.804795 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349162370577-12320-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349162370577-12320-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349162370577-12320-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349162370577-12320-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.61:41161
--webserver_interface=127.12.8.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.61.keytab
--principal=kudu/127.12.8.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:36529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.62:38915,127.12.8.61:41161,127.12.8.60:33203
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:15:59.839574 25786 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.8.61" port: 41161 } attempt: 1
I20260925 15:15:59.840802 25790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:59.840251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.61:41161 (local address 127.0.0.1:44490)
0925 15:15:59.840482 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:59.840635 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.61:41161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260925 15:15:59.841858 25790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:59.841454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.61:41161 (local address 127.0.0.1:44506)
0925 15:15:59.841613 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:59.841698 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.61:41161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260925 15:15:59.842094 25786 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.8.61:41161: Network error: Client connection negotiation failed: client connection to 127.12.8.61:41161: connect: Connection refused (error 111)
I20260925 15:15:59.910409 25786 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.8.61" port: 41161 } attempt: 2
I20260925 15:15:59.911778 25790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:59.911157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.61:41161 (local address 127.0.0.1:44518)
0925 15:15:59.911470 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:59.911603 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.61:41161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260925 15:15:59.912940 25790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:15:59.912433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.61:41161 (local address 127.0.0.1:44528)
0925 15:15:59.912644 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0925 15:15:59.912745 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.61:41161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260925 15:15:59.913146 25786 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.8.61:41161: Network error: Client connection negotiation failed: client connection to 127.12.8.61:41161: connect: Connection refused (error 111)
I20260925 15:16:00.004484 25786 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.8.61" port: 41161 } attempt: 3
I20260925 15:16:00.005951 25790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:00.005378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.61:41161 (local address 127.0.0.1:44540)
0925 15:16:00.005664 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:00.005772 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.61:41161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260925 15:16:00.007082 25790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:00.006648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.61:41161 (local address 127.0.0.1:44552)
0925 15:16:00.006829 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:00.006919 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.61:41161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260925 15:16:00.007275 25786 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.8.61:41161: Network error: Client connection negotiation failed: client connection to 127.12.8.61:41161: connect: Connection refused (error 111)
W20260925 15:16:00.129722 25793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:16:00.130048 25793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:16:00.130249 25793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:16:00.137238 25793 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:16:00.137358 25793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:16:00.137444 25793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:16:00.137550 25793 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:16:00.137633 25793 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:16:00.147177 25793 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:36529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349162370577-12320-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349162370577-12320-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.12.8.62:38915,127.12.8.61:41161,127.12.8.60:33203
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.61.keytab
--principal=kudu/127.12.8.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.61:41161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349162370577-12320-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.12.8.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349162370577-12320-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:16:00.148461 25793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:16:00.149715 25793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 15:16:00.150522 25786 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.8.61" port: 41161 } attempt: 4
I20260925 15:16:00.151715 25790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:00.151127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.61:41161 (local address 127.0.0.1:44558)
0925 15:16:00.151450 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:00.151561 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.61:41161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159}
I20260925 15:16:00.152822 25790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:00.152400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.61:41161 (local address 127.0.0.1:44560)
0925 15:16:00.152562 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:00.152663 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.61:41161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260925 15:16:00.153087 25786 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.8.61:41161: Network error: Client connection negotiation failed: client connection to 127.12.8.61:41161: connect: Connection refused (error 111)
W20260925 15:16:00.162137 25798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:16:00.162089 25799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:16:00.162106 25801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:16:00.167732 25793 server_base.cc:1034] running on GCE node
I20260925 15:16:00.168530 25793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:16:00.169912 25793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:16:00.171149 25793 hybrid_clock.cc:648] HybridClock initialized: now 1790349360171103 us; error 69 us; skew 500 ppm
I20260925 15:16:00.171521 25793 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:16:00 dist-test-slave-p468 krb5kdc[25706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349360, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:16:00.174743 25793 init.cc:377] Logged in from keytab as kudu/127.12.8.61@KRBTEST.COM (short username kudu)
I20260925 15:16:00.176241 25793 webserver.cc:460] Webserver started at http://127.12.8.61:42789/ using document root <none> and password file <none>
I20260925 15:16:00.176870 25793 fs_manager.cc:359] Metadata directory not provided
I20260925 15:16:00.176970 25793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:16:00.177212 25793 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:16:00.180104 25793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349162370577-12320-0/minicluster-data/master-1/data/instance:
uuid: "aa6c1655419d4b21b6912cb9b02436e1"
format_stamp: "Formatted at 2026-09-25 15:16:00 on dist-test-slave-p468"
server_key: "522cfdf21ebbfd931e0c9de50f4289f8"
server_key_iv: "ae287a6e1b31970a1e21c0c2d3cd71a4"
server_key_version: "encryptionkey@0"
I20260925 15:16:00.180830 25793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349162370577-12320-0/minicluster-data/master-1/wal/instance:
uuid: "aa6c1655419d4b21b6912cb9b02436e1"
format_stamp: "Formatted at 2026-09-25 15:16:00 on dist-test-slave-p468"
server_key: "522cfdf21ebbfd931e0c9de50f4289f8"
server_key_iv: "ae287a6e1b31970a1e21c0c2d3cd71a4"
server_key_version: "encryptionkey@0"
I20260925 15:16:00.185746 25793 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260925 15:16:00.189632 25808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:00.190946 25793 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:16:00.191077 25793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349162370577-12320-0/minicluster-data/master-1/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349162370577-12320-0/minicluster-data/master-1/wal
uuid: "aa6c1655419d4b21b6912cb9b02436e1"
format_stamp: "Formatted at 2026-09-25 15:16:00 on dist-test-slave-p468"
server_key: "522cfdf21ebbfd931e0c9de50f4289f8"
server_key_iv: "ae287a6e1b31970a1e21c0c2d3cd71a4"
server_key_version: "encryptionkey@0"
I20260925 15:16:00.191228 25793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349162370577-12320-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349162370577-12320-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349162370577-12320-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:16:00.207702 25793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:16:00.211560 25793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:16:00.211778 25793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:16:00.229594 25793 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.61:41161
I20260925 15:16:00.229625 25859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.61:41161 every 8 connection(s)
I20260925 15:16:00.230929 25793 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349162370577-12320-0/minicluster-data/master-1/data/info.pb
I20260925 15:16:00.234723 25860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:16:00.234861 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 25793
I20260925 15:16:00.235052 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349162370577-12320-0/minicluster-data/master-1/wal/instance
I20260925 15:16:00.235594 12320 external_mini_cluster.cc:1428] Setting key 7806d7d83491d7b93426b7cf2568a3d2
I20260925 15:16:00.244717 25860 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.8.62" port: 38915 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.12.8.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.60@KRBTEST.COM" created.
Sep 25 15:16:00 dist-test-slave-p468 krb5kdc[25706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349360, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.61@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
I20260925 15:16:00.267781 25789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:00.247383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50984 (local address 127.12.8.62:38915)
0925 15:16:00.247561 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:00.247566 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:00.248877 (+ 1311us) server_negotiation.cc:407] Connection header received
0925 15:16:00.250925 (+ 2048us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:00.250930 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:00.251024 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:00.251136 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:00.252596 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:00.253506 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:00.255083 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:00.255535 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:00.255673 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:00.260585 (+ 4912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:16:00.260595 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:16:00.260603 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:16:00.260653 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:16:00.264438 (+ 3785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:00.265199 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:00.265208 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:00.265214 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:00.265364 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:00.265800 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:00.265808 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:00.265814 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:00.266247 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:16:00.266417 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:00.267370 (+ 953us) server_negotiation.cc:299] Negotiation successful
0925 15:16:00.267550 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 15:16:00.269272 25863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:00.247931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:38915 (local address 127.0.0.1:50984)
0925 15:16:00.248718 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:00.248754 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:00.250657 (+ 1903us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:00.251359 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:00.251368 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:00.251871 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:16:00.252411 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:00.252428 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:00.253664 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:00.253668 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:00.254936 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:00.254948 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:00.255761 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:00.255767 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:00.256066 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:00.256826 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:16:00.256853 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:16:00.260338 (+ 3485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:16:00.264612 (+ 4274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:16:00.264619 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:16:00.264632 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:16:00.265062 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:16:00.265475 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:16:00.265478 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:16:00.265497 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:16:00.265679 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:16:00.266370 (+ 691us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:16:00.266375 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:16:00.266910 (+ 535us) client_negotiation.cc:769] Sending connection context
0925 15:16:00.267232 (+ 322us) client_negotiation.cc:240] Negotiation successful
0925 15:16:00.267815 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":349,"threads_started":1}
I20260925 15:16:00.271150 25860 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.8.61" port: 41161 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:16:00 dist-test-slave-p468 krb5kdc[25706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349360, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.61@KRBTEST.COM for kudu/127.12.8.61@KRBTEST.COM
I20260925 15:16:00.286726 25863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:00.271942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.61:41161 (local address 127.0.0.1:44568)
0925 15:16:00.272147 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:00.272167 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:00.272258 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:00.273283 (+ 1025us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:00.273287 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:00.273739 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:16:00.273862 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:00.273870 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:00.275299 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:00.275307 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:00.276130 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:00.276140 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:00.277447 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:00.277452 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:00.277621 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:00.278283 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:16:00.278299 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:16:00.280390 (+ 2091us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:16:00.284054 (+ 3664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:16:00.284059 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:16:00.284066 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:16:00.284467 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:16:00.284885 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:16:00.284892 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:16:00.284896 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:16:00.285033 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:16:00.285638 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:16:00.285646 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:16:00.285822 (+ 176us) client_negotiation.cc:769] Sending connection context
0925 15:16:00.285976 (+ 154us) client_negotiation.cc:240] Negotiation successful
0925 15:16:00.286290 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260925 15:16:00.286850 25865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:00.272142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44568 (local address 127.12.8.61:41161)
0925 15:16:00.272796 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:00.272802 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:00.272825 (+ 23us) server_negotiation.cc:407] Connection header received
0925 15:16:00.272863 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:00.272869 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:00.273045 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:00.273293 (+ 248us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:00.274014 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:00.275152 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:00.276312 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:00.277300 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:00.277459 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:00.280628 (+ 3169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:16:00.280641 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:16:00.280660 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:16:00.280711 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:16:00.283896 (+ 3185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:00.284609 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:00.284621 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:00.284624 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:00.284762 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:00.285146 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:00.285154 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:00.285157 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:00.285507 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:16:00.285751 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:00.286131 (+ 380us) server_negotiation.cc:299] Negotiation successful
0925 15:16:00.286415 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":522,"thread_start_us":386,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.60.keytab.
Entry for principal kudu/127.12.8.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.60.keytab.
I20260925 15:16:00.289397 25860 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.8.60" port: 33203 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:16:00.289947 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349162370577-12320-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349162370577-12320-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349162370577-12320-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349162370577-12320-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.60:33203
--webserver_interface=127.12.8.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.60.keytab
--principal=kudu/127.12.8.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:36529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.62:38915,127.12.8.61:41161,127.12.8.60:33203
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:16:00.290616 25863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:00.289980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.60:33203 (local address 127.0.0.1:45924)
0925 15:16:00.290216 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:00.290386 (+ 170us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.60:33203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260925 15:16:00.290894 25809 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.8.60:33203: connect: Connection refused (error 111)
I20260925 15:16:00.292599 25863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:00.292121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.60:33203 (local address 127.0.0.1:45930)
0925 15:16:00.292322 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:00.292435 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.60:33203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260925 15:16:00.292855 25860 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.8.60:33203: Network error: Client connection negotiation failed: client connection to 127.12.8.60:33203: connect: Connection refused (error 111)
I20260925 15:16:00.342270 25860 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.8.60" port: 33203 } attempt: 1
I20260925 15:16:00.343466 25863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:00.343000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.60:33203 (local address 127.0.0.1:45938)
0925 15:16:00.343203 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:00.343317 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.60:33203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260925 15:16:00.344501 25863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:00.344160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.60:33203 (local address 127.0.0.1:45942)
0925 15:16:00.344324 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:00.344397 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.60:33203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260925 15:16:00.344705 25860 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.8.60:33203: Network error: Client connection negotiation failed: client connection to 127.12.8.60:33203: connect: Connection refused (error 111)
I20260925 15:16:00.412909 25860 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.8.60" port: 33203 } attempt: 2
I20260925 15:16:00.414000 25863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:00.413553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.60:33203 (local address 127.0.0.1:45948)
0925 15:16:00.413745 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:00.413848 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.60:33203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260925 15:16:00.415069 25863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:00.414731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.60:33203 (local address 127.0.0.1:45954)
0925 15:16:00.414895 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:00.414958 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.60:33203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260925 15:16:00.415268 25860 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.8.60:33203: Network error: Client connection negotiation failed: client connection to 127.12.8.60:33203: connect: Connection refused (error 111)
I20260925 15:16:00.452350 25786 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.8.61" port: 41161 } attempt: 5
Sep 25 15:16:00 dist-test-slave-p468 krb5kdc[25706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349359, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.62@KRBTEST.COM for kudu/127.12.8.61@KRBTEST.COM
I20260925 15:16:00.465734 25865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:00.453229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44570 (local address 127.12.8.61:41161)
0925 15:16:00.453412 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:00.453419 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:00.453542 (+ 123us) server_negotiation.cc:407] Connection header received
0925 15:16:00.453773 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:00.453778 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:00.453844 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:00.453958 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:00.454853 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:00.455522 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:00.456790 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:00.457274 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:00.457412 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:00.460481 (+ 3069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:16:00.460486 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:16:00.460489 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:16:00.460520 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:16:00.463194 (+ 2674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:00.463860 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:00.463871 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:00.463875 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:00.463996 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:00.464375 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:00.464380 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:00.464383 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:00.464657 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:16:00.464814 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:00.465196 (+ 382us) server_negotiation.cc:299] Negotiation successful
0925 15:16:00.465410 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 15:16:00.465734 25790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:00.453101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.61:41161 (local address 127.0.0.1:44570)
0925 15:16:00.453412 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:00.453433 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:00.453620 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:00.453988 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:00.453992 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:00.454500 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:16:00.454676 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:00.454685 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:00.455681 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:00.455687 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:00.456651 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:00.456663 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:00.457426 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:00.457432 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:00.457643 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:00.458316 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:16:00.458334 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:16:00.460289 (+ 1955us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:16:00.463334 (+ 3045us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:16:00.463339 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:16:00.463343 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:16:00.463729 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:16:00.464115 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:16:00.464120 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:16:00.464124 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:16:00.464258 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:16:00.464776 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:16:00.464781 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:16:00.464944 (+ 163us) client_negotiation.cc:769] Sending connection context
0925 15:16:00.465090 (+ 146us) client_negotiation.cc:240] Negotiation successful
0925 15:16:00.465374 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166}
I20260925 15:16:00.467133 25786 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.8.60" port: 33203 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:16:00.468106 25790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:00.467727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.60:33203 (local address 127.0.0.1:45968)
0925 15:16:00.467885 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:00.467963 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.60:33203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260925 15:16:00.469038 25790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:00.468725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.60:33203 (local address 127.0.0.1:45972)
0925 15:16:00.468877 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:00.468937 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.60:33203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260925 15:16:00.469261 25786 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.8.60:33203: Network error: Client connection negotiation failed: client connection to 127.12.8.60:33203: connect: Connection refused (error 111)
I20260925 15:16:00.506465 25860 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.8.60" port: 33203 } attempt: 3
I20260925 15:16:00.507627 25863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:00.507117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.60:33203 (local address 127.0.0.1:45976)
0925 15:16:00.507339 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:00.507422 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.60:33203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260925 15:16:00.508733 25863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:00.508309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.60:33203 (local address 127.0.0.1:45980)
0925 15:16:00.508524 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:00.508605 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.60:33203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
W20260925 15:16:00.508935 25860 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.8.60:33203: Network error: Client connection negotiation failed: client connection to 127.12.8.60:33203: connect: Connection refused (error 111)
I20260925 15:16:00.520469 25786 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.8.60" port: 33203 } attempt: 1
I20260925 15:16:00.521634 25790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:00.521066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.60:33203 (local address 127.0.0.1:45992)
0925 15:16:00.521311 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:00.521431 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.60:33203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260925 15:16:00.522816 25790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:00.522388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.60:33203 (local address 127.0.0.1:46002)
0925 15:16:00.522554 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:00.522651 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.60:33203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260925 15:16:00.523046 25786 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.8.60:33203: Network error: Client connection negotiation failed: client connection to 127.12.8.60:33203: connect: Connection refused (error 111)
I20260925 15:16:00.591284 25786 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.8.60" port: 33203 } attempt: 2
I20260925 15:16:00.592466 25790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:00.591914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.60:33203 (local address 127.0.0.1:46012)
0925 15:16:00.592162 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:00.592268 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.60:33203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260925 15:16:00.593482 25790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:00.593113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.60:33203 (local address 127.0.0.1:46018)
0925 15:16:00.593271 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:00.593351 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.60:33203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260925 15:16:00.593694 25786 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.8.60:33203: Network error: Client connection negotiation failed: client connection to 127.12.8.60:33203: connect: Connection refused (error 111)
W20260925 15:16:00.612272 25867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:16:00.612545 25867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:16:00.612640 25867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:16:00.619678 25867 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:16:00.619787 25867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:16:00.619832 25867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:16:00.619880 25867 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:16:00.619926 25867 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:16:00.629640 25867 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:36529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349162370577-12320-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349162370577-12320-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.12.8.62:38915,127.12.8.61:41161,127.12.8.60:33203
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.60.keytab
--principal=kudu/127.12.8.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.60:33203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349162370577-12320-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.12.8.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349162370577-12320-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:16:00.630980 25867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:16:00.632203 25867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:16:00.643740 25875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:16:00.643814 25873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:16:00.643779 25872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:16:00.644733 25867 server_base.cc:1034] running on GCE node
I20260925 15:16:00.645439 25867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:16:00.646905 25867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:16:00.648133 25867 hybrid_clock.cc:648] HybridClock initialized: now 1790349360648099 us; error 50 us; skew 500 ppm
I20260925 15:16:00.648471 25867 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:16:00.652127 25860 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.8.60" port: 33203 } attempt: 4
I20260925 15:16:00.653163 25863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:00.652723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.60:33203 (local address 127.0.0.1:46034)
0925 15:16:00.652942 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:00.653021 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.60:33203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260925 15:16:00.654328 25863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:00.653920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.60:33203 (local address 127.0.0.1:46036)
0925 15:16:00.654096 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:00.654162 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.60:33203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260925 15:16:00.654538 25860 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.8.60:33203: Network error: Client connection negotiation failed: client connection to 127.12.8.60:33203: connect: Connection refused (error 111)
Sep 25 15:16:00 dist-test-slave-p468 krb5kdc[25706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349360, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:16:00.655313 25867 init.cc:377] Logged in from keytab as kudu/127.12.8.60@KRBTEST.COM (short username kudu)
I20260925 15:16:00.656829 25867 webserver.cc:460] Webserver started at http://127.12.8.60:46729/ using document root <none> and password file <none>
I20260925 15:16:00.657570 25867 fs_manager.cc:359] Metadata directory not provided
I20260925 15:16:00.657702 25867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:16:00.658015 25867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:16:00.660954 25867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349162370577-12320-0/minicluster-data/master-2/data/instance:
uuid: "6d21d8a965d84daf9f71e59d505cabbb"
format_stamp: "Formatted at 2026-09-25 15:16:00 on dist-test-slave-p468"
server_key: "404ffccc8ce8270c4225a93f3b9560da"
server_key_iv: "f57ff8362f70e463d34bc91513c3d2bf"
server_key_version: "encryptionkey@0"
I20260925 15:16:00.661761 25867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349162370577-12320-0/minicluster-data/master-2/wal/instance:
uuid: "6d21d8a965d84daf9f71e59d505cabbb"
format_stamp: "Formatted at 2026-09-25 15:16:00 on dist-test-slave-p468"
server_key: "404ffccc8ce8270c4225a93f3b9560da"
server_key_iv: "f57ff8362f70e463d34bc91513c3d2bf"
server_key_version: "encryptionkey@0"
I20260925 15:16:00.666795 25867 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260925 15:16:00.670063 25882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:00.671419 25867 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260925 15:16:00.671602 25867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349162370577-12320-0/minicluster-data/master-2/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349162370577-12320-0/minicluster-data/master-2/wal
uuid: "6d21d8a965d84daf9f71e59d505cabbb"
format_stamp: "Formatted at 2026-09-25 15:16:00 on dist-test-slave-p468"
server_key: "404ffccc8ce8270c4225a93f3b9560da"
server_key_iv: "f57ff8362f70e463d34bc91513c3d2bf"
server_key_version: "encryptionkey@0"
I20260925 15:16:00.671803 25867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349162370577-12320-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349162370577-12320-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349162370577-12320-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:16:00.699918 25786 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.8.60" port: 33203 } attempt: 3
I20260925 15:16:00.700973 25790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:00.700521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.60:33203 (local address 127.0.0.1:46048)
0925 15:16:00.700745 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:00.700821 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.60:33203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260925 15:16:00.701884 25867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:16:00.702116 25790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:00.701672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.60:33203 (local address 127.0.0.1:46056)
0925 15:16:00.701866 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:00.701977 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.60:33203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260925 15:16:00.702364 25786 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.8.60:33203: Network error: Client connection negotiation failed: client connection to 127.12.8.60:33203: connect: Connection refused (error 111)
I20260925 15:16:00.706391 25867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:16:00.706678 25867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:16:00.725016 25867 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.60:33203
I20260925 15:16:00.725052 25933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.60:33203 every 8 connection(s)
I20260925 15:16:00.726336 25867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349162370577-12320-0/minicluster-data/master-2/data/info.pb
I20260925 15:16:00.729787 25934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:16:00.731421 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 25867
I20260925 15:16:00.731616 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349162370577-12320-0/minicluster-data/master-2/wal/instance
I20260925 15:16:00.732234 12320 external_mini_cluster.cc:1428] Setting key 6a65d6e6a6c20d26680f831511bf4af0
I20260925 15:16:00.739889 25934 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.8.62" port: 38915 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:16:00 dist-test-slave-p468 krb5kdc[25706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.60@KRBTEST.COM
Sep 25 15:16:00 dist-test-slave-p468 krb5kdc[25706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349360, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.60@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
I20260925 15:16:00.755432 25937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:00.734263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46060 (local address 127.12.8.60:33203)
0925 15:16:00.735177 (+ 914us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:00.735188 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:00.735252 (+ 64us) server_negotiation.cc:407] Connection header received
0925 15:16:00.736799 (+ 1547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:00.736820 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:00.737207 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:00.737534 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:00.739345 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:00.740592 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:00.742301 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:00.743332 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:00.743497 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:00.747638 (+ 4141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:16:00.747649 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:16:00.747667 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:16:00.747717 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:16:00.751090 (+ 3373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:00.751944 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:00.751952 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:00.751961 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:00.752139 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:00.752591 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:00.752596 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:00.752600 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:00.753078 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:16:00.753653 (+ 575us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:00.753940 (+ 287us) server_negotiation.cc:299] Negotiation successful
0925 15:16:00.754372 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":699,"thread_start_us":472,"threads_started":1}
I20260925 15:16:00.758104 25789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:00.742294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50990 (local address 127.12.8.62:38915)
0925 15:16:00.742505 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:00.742514 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:00.743273 (+ 759us) server_negotiation.cc:407] Connection header received
0925 15:16:00.743437 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:00.743441 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:00.743510 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:00.743590 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:00.744629 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:00.745298 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:00.746631 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:00.747129 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:00.747263 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:00.751482 (+ 4219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:16:00.751488 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:16:00.751492 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:16:00.751530 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:16:00.755102 (+ 3572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:00.755952 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:00.755957 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:00.755960 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:00.756120 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:00.756509 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:00.756513 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:00.756516 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:00.756761 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:16:00.756961 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:00.757505 (+ 544us) server_negotiation.cc:299] Negotiation successful
0925 15:16:00.757724 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 15:16:00.758898 25938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:00.742306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:38915 (local address 127.0.0.1:50990)
0925 15:16:00.743129 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:00.743168 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:00.743319 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:00.743683 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:00.743691 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:00.744223 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:16:00.744469 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:00.744478 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:00.745444 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:00.745449 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:00.746472 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:00.746484 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:00.747343 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:00.747352 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:00.747565 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:00.748332 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:16:00.748361 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:16:00.751285 (+ 2924us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:16:00.755234 (+ 3949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:16:00.755243 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:16:00.755248 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:16:00.755822 (+ 574us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:16:00.756234 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:16:00.756238 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:16:00.756242 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:16:00.756383 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:16:00.756908 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:16:00.756915 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:16:00.757112 (+ 197us) client_negotiation.cc:769] Sending connection context
0925 15:16:00.757336 (+ 224us) client_negotiation.cc:240] Negotiation successful
0925 15:16:00.757640 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":420,"threads_started":1}
I20260925 15:16:00.760162 25934 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.8.61" port: 41161 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:16:00 dist-test-slave-p468 krb5kdc[25706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349360, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.60@KRBTEST.COM for kudu/127.12.8.61@KRBTEST.COM
Sep 25 15:16:00 dist-test-slave-p468 krb5kdc[25706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
I20260925 15:16:00.773904 25938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:00.760869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.61:41161 (local address 127.0.0.1:44586)
0925 15:16:00.761026 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:00.761046 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:00.761133 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:00.761438 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:00.761441 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:00.761730 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:16:00.761849 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:00.761857 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:00.762892 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:00.762897 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:00.764037 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:00.764051 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:00.764960 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:00.764966 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:00.765208 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:00.765689 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:16:00.765715 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:16:00.768186 (+ 2471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:16:00.771326 (+ 3140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:16:00.771331 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:16:00.771335 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:16:00.771829 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:16:00.772203 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:16:00.772208 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:16:00.772212 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:16:00.772411 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:16:00.773008 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:16:00.773030 (+ 22us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:16:00.773189 (+ 159us) client_negotiation.cc:769] Sending connection context
0925 15:16:00.773332 (+ 143us) client_negotiation.cc:240] Negotiation successful
0925 15:16:00.773544 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260925 15:16:00.773937 25865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:00.760979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44586 (local address 127.12.8.61:41161)
0925 15:16:00.761157 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:00.761163 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:00.761182 (+ 19us) server_negotiation.cc:407] Connection header received
0925 15:16:00.761239 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:00.761244 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:00.761316 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:00.761420 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:00.762026 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:00.762748 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:00.764189 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:00.764804 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:00.764910 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:00.768385 (+ 3475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:16:00.768395 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:16:00.768400 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:16:00.768445 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:16:00.771196 (+ 2751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:00.771948 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:00.771952 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:00.771954 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:00.772093 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:00.772539 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:00.772546 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:00.772550 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:00.772888 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:16:00.773048 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:00.773440 (+ 392us) server_negotiation.cc:299] Negotiation successful
0925 15:16:00.773679 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 15:16:00.775161 25934 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.8.60" port: 33203 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:16:00.775277 25789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:00.758984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51002 (local address 127.12.8.62:38915)
0925 15:16:00.759133 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:00.759138 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:00.759306 (+ 168us) server_negotiation.cc:407] Connection header received
0925 15:16:00.759453 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:00.759460 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:00.759527 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:00.759629 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:00.760870 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:00.761839 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:00.763339 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:00.763712 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:00.763801 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:00.769005 (+ 5204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:16:00.769009 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:16:00.769012 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:16:00.769079 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:16:00.772272 (+ 3193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:00.773075 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:00.773080 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:00.773085 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:00.773218 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:00.773620 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:00.773625 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:00.773629 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:00.773976 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:16:00.774274 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:00.774684 (+ 410us) server_negotiation.cc:299] Negotiation successful
0925 15:16:00.774935 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
Sep 25 15:16:00 dist-test-slave-p468 krb5kdc[25706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349360, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.60@KRBTEST.COM for kudu/127.12.8.60@KRBTEST.COM
Sep 25 15:16:00 dist-test-slave-p468 krb5kdc[25706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.61@KRBTEST.COM
I20260925 15:16:00.788754 25938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:00.775858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.60:33203 (local address 127.0.0.1:46072)
0925 15:16:00.776053 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:00.776077 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:00.776171 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:00.776572 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:00.776578 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:00.776983 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:16:00.777151 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:00.777162 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:00.778069 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:00.778076 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:00.779464 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:00.779475 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:00.780357 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:00.780362 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:00.780576 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:00.781081 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:16:00.781102 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:16:00.783541 (+ 2439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:16:00.786587 (+ 3046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:16:00.786595 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:16:00.786599 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:16:00.786943 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:16:00.787301 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:16:00.787304 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:16:00.787306 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:16:00.787430 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:16:00.788017 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:16:00.788023 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:16:00.788183 (+ 160us) client_negotiation.cc:769] Sending connection context
0925 15:16:00.788294 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 15:16:00.788482 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88}
I20260925 15:16:00.789208 25937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:00.776117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46072 (local address 127.12.8.60:33203)
0925 15:16:00.776273 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:00.776278 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:00.776293 (+ 15us) server_negotiation.cc:407] Connection header received
0925 15:16:00.776336 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:00.776341 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:00.776409 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:00.776553 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:00.777306 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:00.777942 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:00.779602 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:00.780226 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:00.780341 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:00.783721 (+ 3380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:16:00.783726 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:16:00.783729 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:16:00.783755 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:16:00.786460 (+ 2705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:00.787058 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:00.787066 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:00.787070 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:00.787207 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:00.787536 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:00.787546 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:00.787549 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:00.787894 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:16:00.788049 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:00.788422 (+ 373us) server_negotiation.cc:299] Negotiation successful
0925 15:16:00.788651 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 15:16:00.792750 25865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:00.777165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44598 (local address 127.12.8.61:41161)
0925 15:16:00.777312 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:00.777318 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:00.777359 (+ 41us) server_negotiation.cc:407] Connection header received
0925 15:16:00.777539 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:00.777544 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:00.777610 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:00.777707 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:00.779249 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:00.780209 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:00.782042 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:00.782609 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:00.782747 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:00.787300 (+ 4553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:16:00.787308 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:16:00.787313 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:16:00.787353 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:16:00.790348 (+ 2995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:00.791095 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:00.791098 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:00.791100 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:00.791224 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:00.791591 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:00.791594 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:00.791596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:00.791831 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:16:00.792018 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:00.792333 (+ 315us) server_negotiation.cc:299] Negotiation successful
0925 15:16:00.792486 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 15:16:00.795050 25934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d21d8a965d84daf9f71e59d505cabbb: Bootstrap starting.
I20260925 15:16:00.797555 12320 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:16:00.798692 25934 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d21d8a965d84daf9f71e59d505cabbb: Neither blocks nor log segments found. Creating new log.
I20260925 15:16:00.799675 25934 log.cc:824] T 00000000000000000000000000000000 P 6d21d8a965d84daf9f71e59d505cabbb: Log is configured to *not* fsync() on all Append() calls
I20260925 15:16:00.802269 25934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d21d8a965d84daf9f71e59d505cabbb: No bootstrap required, opened a new log
I20260925 15:16:00.815495 25934 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d21d8a965d84daf9f71e59d505cabbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89eec0efbc2a4b58a3e2298936464134" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 38915 } } peers { permanent_uuid: "aa6c1655419d4b21b6912cb9b02436e1" member_type: VOTER last_known_addr { host: "127.12.8.61" port: 41161 } } peers { permanent_uuid: "6d21d8a965d84daf9f71e59d505cabbb" member_type: VOTER last_known_addr { host: "127.12.8.60" port: 33203 } }
I20260925 15:16:00.816196 25934 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6d21d8a965d84daf9f71e59d505cabbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:16:00.816418 25934 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d21d8a965d84daf9f71e59d505cabbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d21d8a965d84daf9f71e59d505cabbb, State: Initialized, Role: FOLLOWER
I20260925 15:16:00.817382 25934 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d21d8a965d84daf9f71e59d505cabbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89eec0efbc2a4b58a3e2298936464134" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 38915 } } peers { permanent_uuid: "aa6c1655419d4b21b6912cb9b02436e1" member_type: VOTER last_known_addr { host: "127.12.8.61" port: 41161 } } peers { permanent_uuid: "6d21d8a965d84daf9f71e59d505cabbb" member_type: VOTER last_known_addr { host: "127.12.8.60" port: 33203 } }
I20260925 15:16:00.822539 25789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:00.802736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51008 (local address 127.12.8.62:38915)
0925 15:16:00.802907 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:00.802913 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:00.803086 (+ 173us) server_negotiation.cc:407] Connection header received
0925 15:16:00.803376 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:00.803383 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:00.803463 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:00.803546 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:00.806047 (+ 2501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:00.807223 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:00.809632 (+ 2409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:00.810331 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:00.810461 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:00.814380 (+ 3919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:16:00.814388 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:16:00.814393 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:16:00.814434 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:16:00.818066 (+ 3632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:00.819307 (+ 1241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:00.819316 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:00.819323 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:00.819540 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:00.820172 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:00.820181 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:00.820186 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:00.820664 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:16:00.820969 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:00.821673 (+ 704us) server_negotiation.cc:299] Negotiation successful
0925 15:16:00.821984 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 15:16:00.823042 25865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:00.802390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44610 (local address 127.12.8.61:41161)
0925 15:16:00.802564 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:00.802569 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:00.803574 (+ 1005us) server_negotiation.cc:407] Connection header received
0925 15:16:00.803798 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:00.803803 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:00.803879 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:00.803951 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:00.806417 (+ 2466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:00.807463 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:00.810342 (+ 2879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:00.811039 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:00.811150 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:00.814879 (+ 3729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:16:00.814886 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:16:00.814891 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:16:00.814930 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:16:00.818930 (+ 4000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:00.819988 (+ 1058us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:00.819994 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:00.819999 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:00.820180 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:00.820677 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:00.820685 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:00.820689 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:00.821105 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:16:00.821346 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:00.822404 (+ 1058us) server_negotiation.cc:299] Negotiation successful
0925 15:16:00.822667 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 15:16:00.824084 25947 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d21d8a965d84daf9f71e59d505cabbb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89eec0efbc2a4b58a3e2298936464134" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 38915 } } peers { permanent_uuid: "aa6c1655419d4b21b6912cb9b02436e1" member_type: VOTER last_known_addr { host: "127.12.8.61" port: 41161 } } peers { permanent_uuid: "6d21d8a965d84daf9f71e59d505cabbb" member_type: VOTER last_known_addr { host: "127.12.8.60" port: 33203 } } }
I20260925 15:16:00.824535 25947 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d21d8a965d84daf9f71e59d505cabbb [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:16:00.825304 25934 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d21d8a965d84daf9f71e59d505cabbb [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:16:00.828303 25937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:00.804191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46074 (local address 127.12.8.60:33203)
0925 15:16:00.804333 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:00.804338 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:00.809728 (+ 5390us) server_negotiation.cc:407] Connection header received
0925 15:16:00.809870 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:00.809878 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:00.809958 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:00.810070 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:00.812375 (+ 2305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:00.813596 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:00.815722 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:00.816508 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:00.816611 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:00.820263 (+ 3652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:16:00.820271 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:16:00.820275 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:16:00.820315 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:16:00.825072 (+ 4757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:00.825894 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:00.825900 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:00.825903 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:00.826066 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:00.826490 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:00.826496 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:00.826499 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:00.826901 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:16:00.827193 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:00.827622 (+ 429us) server_negotiation.cc:299] Negotiation successful
0925 15:16:00.827896 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
W20260925 15:16:00.836999 25958 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6d21d8a965d84daf9f71e59d505cabbb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:16:00.837126 25958 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:16:00.833894 25867 master_runner.cc:186] Error getting master registration for 127.12.8.62:38915: OK, instance_id { permanent_uuid: "89eec0efbc2a4b58a3e2298936464134" instance_seqno: 1790349359743247 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 15:16:00.843330 25867 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 15:16:00.879586 25786 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.8.60" port: 33203 } attempt: 4
Sep 25 15:16:00 dist-test-slave-p468 krb5kdc[25706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349359, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.62@KRBTEST.COM for kudu/127.12.8.60@KRBTEST.COM
I20260925 15:16:00.894229 25937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:00.880377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46088 (local address 127.12.8.60:33203)
0925 15:16:00.880580 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:00.880586 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:00.880607 (+ 21us) server_negotiation.cc:407] Connection header received
0925 15:16:00.880694 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:00.880702 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:00.880775 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:00.880875 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:00.881636 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:00.882581 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:00.883895 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:00.884339 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:00.884518 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:00.888678 (+ 4160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:16:00.888683 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:16:00.888686 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:16:00.888716 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:16:00.891516 (+ 2800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:00.892277 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:00.892281 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:00.892283 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:00.892432 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:00.892833 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:00.892837 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:00.892839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:00.893113 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:16:00.893231 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:00.893708 (+ 477us) server_negotiation.cc:299] Negotiation successful
0925 15:16:00.893933 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260925 15:16:00.894304 25790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:00.880238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.60:33203 (local address 127.0.0.1:46088)
0925 15:16:00.880428 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:00.880449 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:00.880560 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:00.880906 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:00.880911 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:00.881328 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:16:00.881490 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:00.881498 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:00.882713 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:00.882722 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:00.883756 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:00.883768 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:00.884551 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:00.884557 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:00.884789 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:00.885430 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:16:00.885456 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:16:00.888496 (+ 3040us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:16:00.891642 (+ 3146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:16:00.891649 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:16:00.891654 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:16:00.892141 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:16:00.892548 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:16:00.892553 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:16:00.892557 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:16:00.892728 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:16:00.893232 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:16:00.893237 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:16:00.893403 (+ 166us) client_negotiation.cc:769] Sending connection context
0925 15:16:00.893593 (+ 190us) client_negotiation.cc:240] Negotiation successful
0925 15:16:00.893887 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260925 15:16:00.899299 25786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89eec0efbc2a4b58a3e2298936464134: Bootstrap starting.
I20260925 15:16:00.902626 25786 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 89eec0efbc2a4b58a3e2298936464134: Neither blocks nor log segments found. Creating new log.
I20260925 15:16:00.903627 25786 log.cc:824] T 00000000000000000000000000000000 P 89eec0efbc2a4b58a3e2298936464134: Log is configured to *not* fsync() on all Append() calls
I20260925 15:16:00.906234 25786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89eec0efbc2a4b58a3e2298936464134: No bootstrap required, opened a new log
I20260925 15:16:00.914338 25786 raft_consensus.cc:348] T 00000000000000000000000000000000 P 89eec0efbc2a4b58a3e2298936464134 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89eec0efbc2a4b58a3e2298936464134" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 38915 } } peers { permanent_uuid: "aa6c1655419d4b21b6912cb9b02436e1" member_type: VOTER last_known_addr { host: "127.12.8.61" port: 41161 } } peers { permanent_uuid: "6d21d8a965d84daf9f71e59d505cabbb" member_type: VOTER last_known_addr { host: "127.12.8.60" port: 33203 } }
I20260925 15:16:00.914639 25786 raft_consensus.cc:374] T 00000000000000000000000000000000 P 89eec0efbc2a4b58a3e2298936464134 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:16:00.914769 25786 raft_consensus.cc:729] T 00000000000000000000000000000000 P 89eec0efbc2a4b58a3e2298936464134 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89eec0efbc2a4b58a3e2298936464134, State: Initialized, Role: FOLLOWER
I20260925 15:16:00.915364 25786 consensus_queue.cc:260] T 00000000000000000000000000000000 P 89eec0efbc2a4b58a3e2298936464134 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89eec0efbc2a4b58a3e2298936464134" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 38915 } } peers { permanent_uuid: "aa6c1655419d4b21b6912cb9b02436e1" member_type: VOTER last_known_addr { host: "127.12.8.61" port: 41161 } } peers { permanent_uuid: "6d21d8a965d84daf9f71e59d505cabbb" member_type: VOTER last_known_addr { host: "127.12.8.60" port: 33203 } }
I20260925 15:16:00.916260 25961 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89eec0efbc2a4b58a3e2298936464134 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89eec0efbc2a4b58a3e2298936464134" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 38915 } } peers { permanent_uuid: "aa6c1655419d4b21b6912cb9b02436e1" member_type: VOTER last_known_addr { host: "127.12.8.61" port: 41161 } } peers { permanent_uuid: "6d21d8a965d84daf9f71e59d505cabbb" member_type: VOTER last_known_addr { host: "127.12.8.60" port: 33203 } } }
I20260925 15:16:00.916668 25961 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89eec0efbc2a4b58a3e2298936464134 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:16:00.917763 25786 sys_catalog.cc:564] T 00000000000000000000000000000000 P 89eec0efbc2a4b58a3e2298936464134 [sys.catalog]: configured and running, proceeding with master startup.
W20260925 15:16:00.939216 25972 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 89eec0efbc2a4b58a3e2298936464134: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:16:00.939452 25972 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:16:00.953738 25860 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.8.60" port: 33203 } attempt: 5
W20260925 15:16:00.959017 25888 tablet.cc:2369] T 00000000000000000000000000000000 P 6d21d8a965d84daf9f71e59d505cabbb: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 15:16:00 dist-test-slave-p468 krb5kdc[25706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349360, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.61@KRBTEST.COM for kudu/127.12.8.60@KRBTEST.COM
I20260925 15:16:00.938059 25719 master_runner.cc:186] Error getting master registration for 127.12.8.61:41161: OK, instance_id { permanent_uuid: "aa6c1655419d4b21b6912cb9b02436e1" instance_seqno: 1790349360224403 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 15:16:00.964293 25719 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 15:16:00.973387 25863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:00.954315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.60:33203 (local address 127.0.0.1:46096)
0925 15:16:00.954566 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:00.954587 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:00.954688 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:00.955318 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:00.955331 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:00.955813 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:16:00.956003 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:00.956015 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:00.957293 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:00.957298 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:00.958507 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:00.958522 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:00.959602 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:00.959607 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:00.959772 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:00.960440 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:16:00.960463 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:16:00.963270 (+ 2807us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:16:00.970137 (+ 6867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:16:00.970144 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:16:00.970149 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:16:00.970755 (+ 606us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:16:00.971239 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:16:00.971245 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:16:00.971252 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:16:00.971430 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:16:00.972249 (+ 819us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:16:00.972255 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:16:00.972429 (+ 174us) client_negotiation.cc:769] Sending connection context
0925 15:16:00.972588 (+ 159us) client_negotiation.cc:240] Negotiation successful
0925 15:16:00.972925 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128}
I20260925 15:16:00.974648 25937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:00.954366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46096 (local address 127.12.8.60:33203)
0925 15:16:00.954904 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:00.954910 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:00.954927 (+ 17us) server_negotiation.cc:407] Connection header received
0925 15:16:00.954977 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:00.954981 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:00.955049 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:00.955131 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:00.956167 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:00.957128 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:00.958673 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:00.959268 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:00.959386 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:00.963518 (+ 4132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:16:00.963524 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:16:00.963528 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:16:00.963563 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:16:00.967374 (+ 3811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:00.970922 (+ 3548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:00.970928 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:00.970931 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:00.971095 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:00.971570 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:00.971576 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:00.971580 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:00.971980 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:16:00.972179 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:00.973926 (+ 1747us) server_negotiation.cc:299] Negotiation successful
0925 15:16:00.974238 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408}
W20260925 15:16:00.982297 25740 tablet.cc:2369] T 00000000000000000000000000000000 P 89eec0efbc2a4b58a3e2298936464134: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:16:00.984362 25860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa6c1655419d4b21b6912cb9b02436e1: Bootstrap starting.
I20260925 15:16:00.988991 25860 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa6c1655419d4b21b6912cb9b02436e1: Neither blocks nor log segments found. Creating new log.
I20260925 15:16:00.992273 25860 log.cc:824] T 00000000000000000000000000000000 P aa6c1655419d4b21b6912cb9b02436e1: Log is configured to *not* fsync() on all Append() calls
I20260925 15:16:00.995893 25860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa6c1655419d4b21b6912cb9b02436e1: No bootstrap required, opened a new log
I20260925 15:16:01.008145 25947 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d21d8a965d84daf9f71e59d505cabbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:16:01.008435 25947 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d21d8a965d84daf9f71e59d505cabbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89eec0efbc2a4b58a3e2298936464134" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 38915 } } peers { permanent_uuid: "aa6c1655419d4b21b6912cb9b02436e1" member_type: VOTER last_known_addr { host: "127.12.8.61" port: 41161 } } peers { permanent_uuid: "6d21d8a965d84daf9f71e59d505cabbb" member_type: VOTER last_known_addr { host: "127.12.8.60" port: 33203 } }
I20260925 15:16:01.010844 25761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6d21d8a965d84daf9f71e59d505cabbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89eec0efbc2a4b58a3e2298936464134" is_pre_election: true
I20260925 15:16:01.011451 25761 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 89eec0efbc2a4b58a3e2298936464134 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d21d8a965d84daf9f71e59d505cabbb in term 0.
I20260925 15:16:01.012104 25947 leader_election.cc:288] T 00000000000000000000000000000000 P 6d21d8a965d84daf9f71e59d505cabbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89eec0efbc2a4b58a3e2298936464134 (127.12.8.62:38915), aa6c1655419d4b21b6912cb9b02436e1 (127.12.8.61:41161)
I20260925 15:16:01.012367 25884 leader_election.cc:302] T 00000000000000000000000000000000 P 6d21d8a965d84daf9f71e59d505cabbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d21d8a965d84daf9f71e59d505cabbb, 89eec0efbc2a4b58a3e2298936464134; no voters:
I20260925 15:16:01.012799 25947 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d21d8a965d84daf9f71e59d505cabbb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:16:01.012926 25947 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d21d8a965d84daf9f71e59d505cabbb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:16:01.013017 25947 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d21d8a965d84daf9f71e59d505cabbb [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:16:01.016702 25947 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d21d8a965d84daf9f71e59d505cabbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89eec0efbc2a4b58a3e2298936464134" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 38915 } } peers { permanent_uuid: "aa6c1655419d4b21b6912cb9b02436e1" member_type: VOTER last_known_addr { host: "127.12.8.61" port: 41161 } } peers { permanent_uuid: "6d21d8a965d84daf9f71e59d505cabbb" member_type: VOTER last_known_addr { host: "127.12.8.60" port: 33203 } }
I20260925 15:16:01.017992 25761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6d21d8a965d84daf9f71e59d505cabbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89eec0efbc2a4b58a3e2298936464134"
I20260925 15:16:01.018303 25761 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 89eec0efbc2a4b58a3e2298936464134 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:16:01.022049 25761 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 89eec0efbc2a4b58a3e2298936464134 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d21d8a965d84daf9f71e59d505cabbb in term 1.
I20260925 15:16:01.022624 25884 leader_election.cc:302] T 00000000000000000000000000000000 P 6d21d8a965d84daf9f71e59d505cabbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d21d8a965d84daf9f71e59d505cabbb, 89eec0efbc2a4b58a3e2298936464134; no voters:
I20260925 15:16:01.023500 25975 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d21d8a965d84daf9f71e59d505cabbb [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:16:01.024295 25947 leader_election.cc:288] T 00000000000000000000000000000000 P 6d21d8a965d84daf9f71e59d505cabbb [CANDIDATE]: Term 1 election: Requested vote from peers 89eec0efbc2a4b58a3e2298936464134 (127.12.8.62:38915), aa6c1655419d4b21b6912cb9b02436e1 (127.12.8.61:41161)
I20260925 15:16:01.028257 25860 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa6c1655419d4b21b6912cb9b02436e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89eec0efbc2a4b58a3e2298936464134" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 38915 } } peers { permanent_uuid: "aa6c1655419d4b21b6912cb9b02436e1" member_type: VOTER last_known_addr { host: "127.12.8.61" port: 41161 } } peers { permanent_uuid: "6d21d8a965d84daf9f71e59d505cabbb" member_type: VOTER last_known_addr { host: "127.12.8.60" port: 33203 } }
I20260925 15:16:01.030862 25860 raft_consensus.cc:374] T 00000000000000000000000000000000 P aa6c1655419d4b21b6912cb9b02436e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:16:01.031195 25860 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa6c1655419d4b21b6912cb9b02436e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa6c1655419d4b21b6912cb9b02436e1, State: Initialized, Role: FOLLOWER
I20260925 15:16:01.032959 25860 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa6c1655419d4b21b6912cb9b02436e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89eec0efbc2a4b58a3e2298936464134" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 38915 } } peers { permanent_uuid: "aa6c1655419d4b21b6912cb9b02436e1" member_type: VOTER last_known_addr { host: "127.12.8.61" port: 41161 } } peers { permanent_uuid: "6d21d8a965d84daf9f71e59d505cabbb" member_type: VOTER last_known_addr { host: "127.12.8.60" port: 33203 } }
I20260925 15:16:01.036193 25975 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d21d8a965d84daf9f71e59d505cabbb [term 1 LEADER]: Becoming Leader. State: Replica: 6d21d8a965d84daf9f71e59d505cabbb, State: Running, Role: LEADER
I20260925 15:16:01.037110 25975 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d21d8a965d84daf9f71e59d505cabbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89eec0efbc2a4b58a3e2298936464134" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 38915 } } peers { permanent_uuid: "aa6c1655419d4b21b6912cb9b02436e1" member_type: VOTER last_known_addr { host: "127.12.8.61" port: 41161 } } peers { permanent_uuid: "6d21d8a965d84daf9f71e59d505cabbb" member_type: VOTER last_known_addr { host: "127.12.8.60" port: 33203 } }
I20260925 15:16:01.042793 25977 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa6c1655419d4b21b6912cb9b02436e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89eec0efbc2a4b58a3e2298936464134" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 38915 } } peers { permanent_uuid: "aa6c1655419d4b21b6912cb9b02436e1" member_type: VOTER last_known_addr { host: "127.12.8.61" port: 41161 } } peers { permanent_uuid: "6d21d8a965d84daf9f71e59d505cabbb" member_type: VOTER last_known_addr { host: "127.12.8.60" port: 33203 } } }
I20260925 15:16:01.043243 25977 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa6c1655419d4b21b6912cb9b02436e1 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:16:01.045605 25835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6d21d8a965d84daf9f71e59d505cabbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa6c1655419d4b21b6912cb9b02436e1" is_pre_election: true
I20260925 15:16:01.046049 25860 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa6c1655419d4b21b6912cb9b02436e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:16:01.046294 25835 raft_consensus.cc:2454] T 00000000000000000000000000000000 P aa6c1655419d4b21b6912cb9b02436e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d21d8a965d84daf9f71e59d505cabbb in term 0.
I20260925 15:16:01.045684 25834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6d21d8a965d84daf9f71e59d505cabbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa6c1655419d4b21b6912cb9b02436e1"
I20260925 15:16:01.046816 25834 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa6c1655419d4b21b6912cb9b02436e1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:16:01.050376 25834 raft_consensus.cc:2454] T 00000000000000000000000000000000 P aa6c1655419d4b21b6912cb9b02436e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d21d8a965d84daf9f71e59d505cabbb in term 1.
I20260925 15:16:01.059875 25947 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d21d8a965d84daf9f71e59d505cabbb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d21d8a965d84daf9f71e59d505cabbb. Latest consensus state: current_term: 1 leader_uuid: "6d21d8a965d84daf9f71e59d505cabbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89eec0efbc2a4b58a3e2298936464134" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 38915 } } peers { permanent_uuid: "aa6c1655419d4b21b6912cb9b02436e1" member_type: VOTER last_known_addr { host: "127.12.8.61" port: 41161 } } peers { permanent_uuid: "6d21d8a965d84daf9f71e59d505cabbb" member_type: VOTER last_known_addr { host: "127.12.8.60" port: 33203 } } }
I20260925 15:16:01.060324 25947 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d21d8a965d84daf9f71e59d505cabbb [sys.catalog]: This master's current role is: LEADER
I20260925 15:16:01.061512 25980 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:16:01.064162 25980 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:16:01.088390 25834 raft_consensus.cc:1261] T 00000000000000000000000000000000 P aa6c1655419d4b21b6912cb9b02436e1 [term 1 FOLLOWER]: Refusing update from remote peer 6d21d8a965d84daf9f71e59d505cabbb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:16:01.089869 25947 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6d21d8a965d84daf9f71e59d505cabbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa6c1655419d4b21b6912cb9b02436e1" member_type: VOTER last_known_addr { host: "127.12.8.61" port: 41161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:16:01.099249 25761 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 89eec0efbc2a4b58a3e2298936464134 [term 1 FOLLOWER]: Refusing update from remote peer 6d21d8a965d84daf9f71e59d505cabbb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:16:01.100595 25947 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6d21d8a965d84daf9f71e59d505cabbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "89eec0efbc2a4b58a3e2298936464134" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 38915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:16:01.123654 25977 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa6c1655419d4b21b6912cb9b02436e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d21d8a965d84daf9f71e59d505cabbb. Latest consensus state: current_term: 1 leader_uuid: "6d21d8a965d84daf9f71e59d505cabbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89eec0efbc2a4b58a3e2298936464134" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 38915 } } peers { permanent_uuid: "aa6c1655419d4b21b6912cb9b02436e1" member_type: VOTER last_known_addr { host: "127.12.8.61" port: 41161 } } peers { permanent_uuid: "6d21d8a965d84daf9f71e59d505cabbb" member_type: VOTER last_known_addr { host: "127.12.8.60" port: 33203 } } }
I20260925 15:16:01.124015 25977 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa6c1655419d4b21b6912cb9b02436e1 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:16:01.126575 25982 mvcc.cc:205] Tried to move back new op lower bound from 7333270983002697728 to 7333270982817255424. Current Snapshot: MvccSnapshot[applied={T|T < 7333270983002697728}]
I20260925 15:16:01.127084 25986 mvcc.cc:205] Tried to move back new op lower bound from 7333270983002697728 to 7333270982817255424. Current Snapshot: MvccSnapshot[applied={T|T < 7333270983002697728}]
I20260925 15:16:01.133194 25961 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89eec0efbc2a4b58a3e2298936464134 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d21d8a965d84daf9f71e59d505cabbb. Latest consensus state: current_term: 1 leader_uuid: "6d21d8a965d84daf9f71e59d505cabbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89eec0efbc2a4b58a3e2298936464134" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 38915 } } peers { permanent_uuid: "aa6c1655419d4b21b6912cb9b02436e1" member_type: VOTER last_known_addr { host: "127.12.8.61" port: 41161 } } peers { permanent_uuid: "6d21d8a965d84daf9f71e59d505cabbb" member_type: VOTER last_known_addr { host: "127.12.8.60" port: 33203 } } }
I20260925 15:16:01.133548 25961 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89eec0efbc2a4b58a3e2298936464134 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:16:01.136374 25977 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa6c1655419d4b21b6912cb9b02436e1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6d21d8a965d84daf9f71e59d505cabbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89eec0efbc2a4b58a3e2298936464134" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 38915 } } peers { permanent_uuid: "aa6c1655419d4b21b6912cb9b02436e1" member_type: VOTER last_known_addr { host: "127.12.8.61" port: 41161 } } peers { permanent_uuid: "6d21d8a965d84daf9f71e59d505cabbb" member_type: VOTER last_known_addr { host: "127.12.8.60" port: 33203 } } }
I20260925 15:16:01.136622 25977 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa6c1655419d4b21b6912cb9b02436e1 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:16:01.137462 25987 mvcc.cc:205] Tried to move back new op lower bound from 7333270983002697728 to 7333270982817255424. Current Snapshot: MvccSnapshot[applied={T|T < 7333270983002697728}]
I20260925 15:16:01.138222 25947 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d21d8a965d84daf9f71e59d505cabbb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6d21d8a965d84daf9f71e59d505cabbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89eec0efbc2a4b58a3e2298936464134" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 38915 } } peers { permanent_uuid: "aa6c1655419d4b21b6912cb9b02436e1" member_type: VOTER last_known_addr { host: "127.12.8.61" port: 41161 } } peers { permanent_uuid: "6d21d8a965d84daf9f71e59d505cabbb" member_type: VOTER last_known_addr { host: "127.12.8.60" port: 33203 } } }
I20260925 15:16:01.138541 25947 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d21d8a965d84daf9f71e59d505cabbb [sys.catalog]: This master's current role is: LEADER
I20260925 15:16:01.138837 25947 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d21d8a965d84daf9f71e59d505cabbb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6d21d8a965d84daf9f71e59d505cabbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89eec0efbc2a4b58a3e2298936464134" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 38915 } } peers { permanent_uuid: "aa6c1655419d4b21b6912cb9b02436e1" member_type: VOTER last_known_addr { host: "127.12.8.61" port: 41161 } } peers { permanent_uuid: "6d21d8a965d84daf9f71e59d505cabbb" member_type: VOTER last_known_addr { host: "127.12.8.60" port: 33203 } } }
I20260925 15:16:01.139108 25947 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d21d8a965d84daf9f71e59d505cabbb [sys.catalog]: This master's current role is: LEADER
I20260925 15:16:01.143438 25980 catalog_manager.cc:1347] Generated new cluster ID: e8041e400beb461eadb51340861050a2
I20260925 15:16:01.143584 25980 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:16:01.153393 25961 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89eec0efbc2a4b58a3e2298936464134 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6d21d8a965d84daf9f71e59d505cabbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89eec0efbc2a4b58a3e2298936464134" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 38915 } } peers { permanent_uuid: "aa6c1655419d4b21b6912cb9b02436e1" member_type: VOTER last_known_addr { host: "127.12.8.61" port: 41161 } } peers { permanent_uuid: "6d21d8a965d84daf9f71e59d505cabbb" member_type: VOTER last_known_addr { host: "127.12.8.60" port: 33203 } } }
I20260925 15:16:01.153658 25961 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89eec0efbc2a4b58a3e2298936464134 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:16:01.183688 25980 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:16:01.185617 25980 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:16:01.193415 25998 catalog_manager.cc:1259] Loaded cluster ID: e8041e400beb461eadb51340861050a2
I20260925 15:16:01.193598 25998 catalog_manager.cc:1552] T 00000000000000000000000000000000 P aa6c1655419d4b21b6912cb9b02436e1: loading cluster ID for follower catalog manager: success
I20260925 15:16:01.197417 25998 catalog_manager.cc:1574] T 00000000000000000000000000000000 P aa6c1655419d4b21b6912cb9b02436e1: acquiring CA information for follower catalog manager: success
W20260925 15:16:01.198786 25998 catalog_manager.cc:873] Not found: no TSK found in the system table: failed to prepare follower catalog manager, will retry
I20260925 15:16:01.208726 25980 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6d21d8a965d84daf9f71e59d505cabbb: Generated new TSK 0
I20260925 15:16:01.209740 25980 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:16:01.941428 25972 catalog_manager.cc:1259] Loaded cluster ID: e8041e400beb461eadb51340861050a2
I20260925 15:16:01.941606 25972 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 89eec0efbc2a4b58a3e2298936464134: loading cluster ID for follower catalog manager: success
I20260925 15:16:01.944088 25972 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 89eec0efbc2a4b58a3e2298936464134: acquiring CA information for follower catalog manager: success
I20260925 15:16:01.945837 25972 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 89eec0efbc2a4b58a3e2298936464134: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 15:16:02.201478 25998 catalog_manager.cc:1602] T 00000000000000000000000000000000 P aa6c1655419d4b21b6912cb9b02436e1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 15:16:02.361145 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 25719
I20260925 15:16:02.362371 25781 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:16:02.472610 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 25719
I20260925 15:16:02.493085 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 25793
I20260925 15:16:02.494344 25855 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:16:02.600703 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 25793
I20260925 15:16:02.620589 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 25867
I20260925 15:16:02.621685 25929 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:16:02.731334 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 25867
2026-09-25T15:16:02Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4765 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349162370577-12320-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:16:02 dist-test-slave-p468 krb5kdc[26010](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:16:02 dist-test-slave-p468 krb5kdc[26010](info): set up 2 sockets
Sep 25 15:16:02 dist-test-slave-p468 krb5kdc[26010](info): commencing operation
krb5kdc: starting...
W20260925 15:16:03.826673 12320 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349162370577-12320-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349162370577-12320-0/krb5kdc/test-user.keytab.
Sep 25 15:16:03 dist-test-slave-p468 krb5kdc[26010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349363, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:16:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:16:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.8.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab.
Entry for principal kudu/127.12.8.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab.
I20260925 15:16:04.057662 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349162370577-12320-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349162370577-12320-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349162370577-12320-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.62:46191
--webserver_interface=127.12.8.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab
--principal=kudu/127.12.8.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:38831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.62:46191
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:16:04.380486 26023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:16:04.380775 26023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:16:04.380867 26023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:16:04.387811 26023 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:16:04.387908 26023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:16:04.387966 26023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:16:04.388026 26023 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:16:04.388079 26023 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:16:04.397501 26023 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:38831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349162370577-12320-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349162370577-12320-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.8.62:46191
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab
--principal=kudu/127.12.8.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.62:46191
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.8.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349162370577-12320-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:16:04.398784 26023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:16:04.400034 26023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:16:04.411926 26028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:16:04.411844 26031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:16:04.411926 26029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:16:04.413234 26023 server_base.cc:1034] running on GCE node
I20260925 15:16:04.414042 26023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:16:04.415515 26023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:16:04.416728 26023 hybrid_clock.cc:648] HybridClock initialized: now 1790349364416672 us; error 44 us; skew 500 ppm
I20260925 15:16:04.417092 26023 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:16:04 dist-test-slave-p468 krb5kdc[26010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349364, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:16:04.420338 26023 init.cc:377] Logged in from keytab as kudu/127.12.8.62@KRBTEST.COM (short username kudu)
I20260925 15:16:04.421927 26023 webserver.cc:460] Webserver started at http://127.12.8.62:39789/ using document root <none> and password file <none>
I20260925 15:16:04.422700 26023 fs_manager.cc:359] Metadata directory not provided
I20260925 15:16:04.422832 26023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:16:04.423153 26023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:16:04.426491 26023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349162370577-12320-0/minicluster-data/master-0/data/instance:
uuid: "34d86cb0cbe946ac84e930f526d50f5c"
format_stamp: "Formatted at 2026-09-25 15:16:04 on dist-test-slave-p468"
server_key: "fb642f9ed838611686d2a2b29c2e7cde"
server_key_iv: "628c5c6a880f3331d1be8c59485e9d07"
server_key_version: "encryptionkey@0"
I20260925 15:16:04.427291 26023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349162370577-12320-0/minicluster-data/master-0/wal/instance:
uuid: "34d86cb0cbe946ac84e930f526d50f5c"
format_stamp: "Formatted at 2026-09-25 15:16:04 on dist-test-slave-p468"
server_key: "fb642f9ed838611686d2a2b29c2e7cde"
server_key_iv: "628c5c6a880f3331d1be8c59485e9d07"
server_key_version: "encryptionkey@0"
I20260925 15:16:04.432621 26023 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 15:16:04.436127 26038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:04.437402 26023 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 15:16:04.437592 26023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349162370577-12320-0/minicluster-data/master-0/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349162370577-12320-0/minicluster-data/master-0/wal
uuid: "34d86cb0cbe946ac84e930f526d50f5c"
format_stamp: "Formatted at 2026-09-25 15:16:04 on dist-test-slave-p468"
server_key: "fb642f9ed838611686d2a2b29c2e7cde"
server_key_iv: "628c5c6a880f3331d1be8c59485e9d07"
server_key_version: "encryptionkey@0"
I20260925 15:16:04.437803 26023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349162370577-12320-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349162370577-12320-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349162370577-12320-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:16:04.453946 26023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:16:04.458290 26023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:16:04.458575 26023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:16:04.478762 26023 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.62:46191
I20260925 15:16:04.478785 26089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.62:46191 every 8 connection(s)
I20260925 15:16:04.480216 26023 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
I20260925 15:16:04.484302 26090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:16:04.488140 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 26023
I20260925 15:16:04.488384 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349162370577-12320-0/minicluster-data/master-0/wal/instance
I20260925 15:16:04.489059 12320 external_mini_cluster.cc:1428] Setting key d14e05b4f2124b3cacf88898b60456f4
I20260925 15:16:04.495105 26090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34d86cb0cbe946ac84e930f526d50f5c: Bootstrap starting.
I20260925 15:16:04.498664 26090 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 34d86cb0cbe946ac84e930f526d50f5c: Neither blocks nor log segments found. Creating new log.
I20260925 15:16:04.499812 26090 log.cc:824] T 00000000000000000000000000000000 P 34d86cb0cbe946ac84e930f526d50f5c: Log is configured to *not* fsync() on all Append() calls
I20260925 15:16:04.503190 26090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34d86cb0cbe946ac84e930f526d50f5c: No bootstrap required, opened a new log
Sep 25 15:16:04 dist-test-slave-p468 krb5kdc[26010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349363, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
I20260925 15:16:04.510864 26090 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34d86cb0cbe946ac84e930f526d50f5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34d86cb0cbe946ac84e930f526d50f5c" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 46191 } }
I20260925 15:16:04.511240 26090 raft_consensus.cc:374] T 00000000000000000000000000000000 P 34d86cb0cbe946ac84e930f526d50f5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:16:04.511366 26090 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34d86cb0cbe946ac84e930f526d50f5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34d86cb0cbe946ac84e930f526d50f5c, State: Initialized, Role: FOLLOWER
I20260925 15:16:04.512033 26090 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34d86cb0cbe946ac84e930f526d50f5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34d86cb0cbe946ac84e930f526d50f5c" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 46191 } }
I20260925 15:16:04.512291 26090 raft_consensus.cc:388] T 00000000000000000000000000000000 P 34d86cb0cbe946ac84e930f526d50f5c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:16:04.512434 26090 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34d86cb0cbe946ac84e930f526d50f5c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:16:04.512621 26090 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34d86cb0cbe946ac84e930f526d50f5c [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:16:04.512702 26093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:04.490821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42702 (local address 127.12.8.62:46191)
0925 15:16:04.491611 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:04.491633 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:04.491679 (+ 46us) server_negotiation.cc:407] Connection header received
0925 15:16:04.493631 (+ 1952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:04.493654 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:04.494028 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:04.494422 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:04.496271 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:04.497558 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:04.499495 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:04.500242 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:04.500452 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:04.505250 (+ 4798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:16:04.505257 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:16:04.505270 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:16:04.505311 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:16:04.508472 (+ 3161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:04.509192 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:04.509198 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:04.509205 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:04.509368 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:04.509794 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:04.509798 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:04.509800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:04.510378 (+ 578us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:16:04.510702 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:04.511145 (+ 443us) server_negotiation.cc:299] Negotiation successful
0925 15:16:04.511592 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":385,"threads_started":1}
I20260925 15:16:04.515606 26090 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34d86cb0cbe946ac84e930f526d50f5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34d86cb0cbe946ac84e930f526d50f5c" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 46191 } }
I20260925 15:16:04.516103 26090 leader_election.cc:302] T 00000000000000000000000000000000 P 34d86cb0cbe946ac84e930f526d50f5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 34d86cb0cbe946ac84e930f526d50f5c; no voters:
I20260925 15:16:04.516599 12320 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:16:04.516788 26090 leader_election.cc:288] T 00000000000000000000000000000000 P 34d86cb0cbe946ac84e930f526d50f5c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:16:04.516881 26095 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34d86cb0cbe946ac84e930f526d50f5c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:16:04.517984 26095 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34d86cb0cbe946ac84e930f526d50f5c [term 1 LEADER]: Becoming Leader. State: Replica: 34d86cb0cbe946ac84e930f526d50f5c, State: Running, Role: LEADER
I20260925 15:16:04.518600 26095 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34d86cb0cbe946ac84e930f526d50f5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34d86cb0cbe946ac84e930f526d50f5c" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 46191 } }
I20260925 15:16:04.519102 26090 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34d86cb0cbe946ac84e930f526d50f5c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:16:04.521737 26096 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34d86cb0cbe946ac84e930f526d50f5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "34d86cb0cbe946ac84e930f526d50f5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34d86cb0cbe946ac84e930f526d50f5c" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 46191 } } }
I20260925 15:16:04.522169 26096 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34d86cb0cbe946ac84e930f526d50f5c [sys.catalog]: This master's current role is: LEADER
I20260925 15:16:04.522707 26097 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34d86cb0cbe946ac84e930f526d50f5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34d86cb0cbe946ac84e930f526d50f5c. Latest consensus state: current_term: 1 leader_uuid: "34d86cb0cbe946ac84e930f526d50f5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34d86cb0cbe946ac84e930f526d50f5c" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 46191 } } }
I20260925 15:16:04.522997 26097 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34d86cb0cbe946ac84e930f526d50f5c [sys.catalog]: This master's current role is: LEADER
I20260925 15:16:04.527235 26104 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:16:04.531270 26104 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:16:04.539808 26104 catalog_manager.cc:1347] Generated new cluster ID: cd7821f7ecf54d43874dcaf1c6f47856
I20260925 15:16:04.539921 26104 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:16:04.569293 26104 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:16:04.570761 26104 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:16:04.578325 26104 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 34d86cb0cbe946ac84e930f526d50f5c: Generated new TSK 0
I20260925 15:16:04.579208 26104 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:16:04.629560 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 26023
I20260925 15:16:04.630795 26085 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:16:04.727497 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 26023
2026-09-25T15:16:04Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1989 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:16:04 dist-test-slave-p468 krb5kdc[26118](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:16:04 dist-test-slave-p468 krb5kdc[26118](info): set up 2 sockets
Sep 25 15:16:04 dist-test-slave-p468 krb5kdc[26118](info): commencing operation
krb5kdc: starting...
W20260925 15:16:05.823004 12320 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.001s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/krb5kdc/test-user.keytab.
Sep 25 15:16:05 dist-test-slave-p468 krb5kdc[26118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:16:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:16:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.8.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab.
Entry for principal kudu/127.12.8.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab.
I20260925 15:16:06.069878 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.62:33201
--webserver_interface=127.12.8.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab
--principal=kudu/127.12.8.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:35145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.62:33201
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:16:06.388820 26131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:16:06.389166 26131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:16:06.389355 26131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:16:06.396713 26131 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:16:06.396816 26131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:16:06.396891 26131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:16:06.397010 26131 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:16:06.397068 26131 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:16:06.407770 26131 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:35145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.8.62:33201
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab
--principal=kudu/127.12.8.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.62:33201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.8.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:16:06.409082 26131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:16:06.410468 26131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:16:06.421749 26136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:16:06.421661 26137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:16:06.421952 26139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:16:06.422690 26131 server_base.cc:1034] running on GCE node
I20260925 15:16:06.423549 26131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:16:06.425004 26131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:16:06.426293 26131 hybrid_clock.cc:648] HybridClock initialized: now 1790349366426240 us; error 34 us; skew 500 ppm
I20260925 15:16:06.426679 26131 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:16:06 dist-test-slave-p468 krb5kdc[26118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349366, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:16:06.429741 26131 init.cc:377] Logged in from keytab as kudu/127.12.8.62@KRBTEST.COM (short username kudu)
I20260925 15:16:06.431229 26131 webserver.cc:460] Webserver started at http://127.12.8.62:46131/ using document root <none> and password file <none>
I20260925 15:16:06.431870 26131 fs_manager.cc:359] Metadata directory not provided
I20260925 15:16:06.431998 26131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:16:06.432309 26131 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:16:06.435223 26131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/master-0/data/instance:
uuid: "594bdcb833214f8ca0ecc6906c3c8e64"
format_stamp: "Formatted at 2026-09-25 15:16:06 on dist-test-slave-p468"
server_key: "21b8a2d0a9e431406afae1549ce7fdac"
server_key_iv: "2d27c4eb28a0816f2bd06cb6ba3eea4d"
server_key_version: "encryptionkey@0"
I20260925 15:16:06.435954 26131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/master-0/wal/instance:
uuid: "594bdcb833214f8ca0ecc6906c3c8e64"
format_stamp: "Formatted at 2026-09-25 15:16:06 on dist-test-slave-p468"
server_key: "21b8a2d0a9e431406afae1549ce7fdac"
server_key_iv: "2d27c4eb28a0816f2bd06cb6ba3eea4d"
server_key_version: "encryptionkey@0"
I20260925 15:16:06.440856 26131 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 15:16:06.444145 26146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:06.445461 26131 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 15:16:06.445667 26131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/master-0/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/master-0/wal
uuid: "594bdcb833214f8ca0ecc6906c3c8e64"
format_stamp: "Formatted at 2026-09-25 15:16:06 on dist-test-slave-p468"
server_key: "21b8a2d0a9e431406afae1549ce7fdac"
server_key_iv: "2d27c4eb28a0816f2bd06cb6ba3eea4d"
server_key_version: "encryptionkey@0"
I20260925 15:16:06.445861 26131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:16:06.460848 26131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:16:06.464751 26131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:16:06.465037 26131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:16:06.485297 26131 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.62:33201
I20260925 15:16:06.485319 26197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.62:33201 every 8 connection(s)
I20260925 15:16:06.486862 26131 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
I20260925 15:16:06.490828 26198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:16:06.491269 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 26131
I20260925 15:16:06.491492 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/master-0/wal/instance
I20260925 15:16:06.492333 12320 external_mini_cluster.cc:1428] Setting key 0b9288fa83ce1b6a40d0cb7eb6cdd786
I20260925 15:16:06.505169 26198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 594bdcb833214f8ca0ecc6906c3c8e64: Bootstrap starting.
Sep 25 15:16:06 dist-test-slave-p468 krb5kdc[26118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
I20260925 15:16:06.508802 26198 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 594bdcb833214f8ca0ecc6906c3c8e64: Neither blocks nor log segments found. Creating new log.
I20260925 15:16:06.509836 26198 log.cc:824] T 00000000000000000000000000000000 P 594bdcb833214f8ca0ecc6906c3c8e64: Log is configured to *not* fsync() on all Append() calls
I20260925 15:16:06.512576 26198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 594bdcb833214f8ca0ecc6906c3c8e64: No bootstrap required, opened a new log
I20260925 15:16:06.515977 26201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:06.494506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53354 (local address 127.12.8.62:33201)
0925 15:16:06.495334 (+ 828us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:06.495345 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:06.495385 (+ 40us) server_negotiation.cc:407] Connection header received
0925 15:16:06.497306 (+ 1921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:06.497329 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:06.497731 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:06.498083 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:06.499892 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:06.501190 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:06.503224 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:06.503872 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:06.503996 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:06.508812 (+ 4816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:16:06.508821 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:16:06.508835 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:16:06.508898 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:16:06.511984 (+ 3086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:06.512655 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:06.512661 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:06.512666 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:06.512851 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:06.513255 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:06.513260 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:06.513263 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:06.513759 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:16:06.514092 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:06.514626 (+ 534us) server_negotiation.cc:299] Negotiation successful
0925 15:16:06.515028 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":373,"threads_started":1}
I20260925 15:16:06.521121 26198 raft_consensus.cc:348] T 00000000000000000000000000000000 P 594bdcb833214f8ca0ecc6906c3c8e64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "594bdcb833214f8ca0ecc6906c3c8e64" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 33201 } }
I20260925 15:16:06.521482 26198 raft_consensus.cc:374] T 00000000000000000000000000000000 P 594bdcb833214f8ca0ecc6906c3c8e64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:16:06.521586 26198 raft_consensus.cc:729] T 00000000000000000000000000000000 P 594bdcb833214f8ca0ecc6906c3c8e64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 594bdcb833214f8ca0ecc6906c3c8e64, State: Initialized, Role: FOLLOWER
I20260925 15:16:06.522105 26198 consensus_queue.cc:260] T 00000000000000000000000000000000 P 594bdcb833214f8ca0ecc6906c3c8e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "594bdcb833214f8ca0ecc6906c3c8e64" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 33201 } }
I20260925 15:16:06.522405 26198 raft_consensus.cc:388] T 00000000000000000000000000000000 P 594bdcb833214f8ca0ecc6906c3c8e64 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:16:06.522541 26198 raft_consensus.cc:482] T 00000000000000000000000000000000 P 594bdcb833214f8ca0ecc6906c3c8e64 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:16:06.522719 26198 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 594bdcb833214f8ca0ecc6906c3c8e64 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:16:06.524780 26198 raft_consensus.cc:504] T 00000000000000000000000000000000 P 594bdcb833214f8ca0ecc6906c3c8e64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "594bdcb833214f8ca0ecc6906c3c8e64" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 33201 } }
I20260925 15:16:06.525249 26198 leader_election.cc:302] T 00000000000000000000000000000000 P 594bdcb833214f8ca0ecc6906c3c8e64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 594bdcb833214f8ca0ecc6906c3c8e64; no voters:
I20260925 15:16:06.525913 26198 leader_election.cc:288] T 00000000000000000000000000000000 P 594bdcb833214f8ca0ecc6906c3c8e64 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:16:06.526052 26203 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 594bdcb833214f8ca0ecc6906c3c8e64 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:16:06.527097 26203 raft_consensus.cc:686] T 00000000000000000000000000000000 P 594bdcb833214f8ca0ecc6906c3c8e64 [term 1 LEADER]: Becoming Leader. State: Replica: 594bdcb833214f8ca0ecc6906c3c8e64, State: Running, Role: LEADER
I20260925 15:16:06.527640 26203 consensus_queue.cc:237] T 00000000000000000000000000000000 P 594bdcb833214f8ca0ecc6906c3c8e64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "594bdcb833214f8ca0ecc6906c3c8e64" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 33201 } }
I20260925 15:16:06.528218 26198 sys_catalog.cc:564] T 00000000000000000000000000000000 P 594bdcb833214f8ca0ecc6906c3c8e64 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:16:06.531505 26205 sys_catalog.cc:455] T 00000000000000000000000000000000 P 594bdcb833214f8ca0ecc6906c3c8e64 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 594bdcb833214f8ca0ecc6906c3c8e64. Latest consensus state: current_term: 1 leader_uuid: "594bdcb833214f8ca0ecc6906c3c8e64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "594bdcb833214f8ca0ecc6906c3c8e64" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 33201 } } }
I20260925 15:16:06.531900 26204 sys_catalog.cc:455] T 00000000000000000000000000000000 P 594bdcb833214f8ca0ecc6906c3c8e64 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "594bdcb833214f8ca0ecc6906c3c8e64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "594bdcb833214f8ca0ecc6906c3c8e64" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 33201 } } }
I20260925 15:16:06.532403 26204 sys_catalog.cc:458] T 00000000000000000000000000000000 P 594bdcb833214f8ca0ecc6906c3c8e64 [sys.catalog]: This master's current role is: LEADER
I20260925 15:16:06.532699 26205 sys_catalog.cc:458] T 00000000000000000000000000000000 P 594bdcb833214f8ca0ecc6906c3c8e64 [sys.catalog]: This master's current role is: LEADER
I20260925 15:16:06.533675 26212 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:16:06.538822 26212 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:16:06.547605 26212 catalog_manager.cc:1347] Generated new cluster ID: d0808ddb0aa64a32a4f15a30c19c15c3
I20260925 15:16:06.547714 26212 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:16:06.563482 26212 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:16:06.565263 26212 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:16:06.572222 26212 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 594bdcb833214f8ca0ecc6906c3c8e64: Generated new TSK 0
I20260925 15:16:06.573113 26212 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.8.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.1.keytab.
Entry for principal kudu/127.12.8.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.1.keytab.
I20260925 15:16:06.629599 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.1:0
--local_ip_for_outbound_sockets=127.12.8.1
--webserver_interface=127.12.8.1
--webserver_port=0
--tserver_master_addrs=127.12.8.62:33201
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.1.keytab
--principal=kudu/127.12.8.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:35145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:16:06.959630 26224 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:16:06.959939 26224 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:16:06.960106 26224 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:16:06.967324 26224 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:16:06.967471 26224 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:16:06.967654 26224 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.1
I20260925 15:16:06.977154 26224 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:35145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.1.keytab
--principal=kudu/127.12.8.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.8.1
--webserver_port=0
--tserver_master_addrs=127.12.8.62:33201
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.1
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:16:06.978374 26224 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:16:06.979754 26224 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:16:06.992630 26230 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:16:06.993362 26229 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:16:06.993388 26224 server_base.cc:1034] running on GCE node
W20260925 15:16:06.993491 26232 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:16:06.994107 26224 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:16:06.995042 26224 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:16:06.996249 26224 hybrid_clock.cc:648] HybridClock initialized: now 1790349366996204 us; error 32 us; skew 500 ppm
I20260925 15:16:06.996559 26224 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:16:06 dist-test-slave-p468 krb5kdc[26118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349366, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:16:06.999724 26224 init.cc:377] Logged in from keytab as kudu/127.12.8.1@KRBTEST.COM (short username kudu)
I20260925 15:16:07.001061 26224 webserver.cc:460] Webserver started at http://127.12.8.1:38723/ using document root <none> and password file <none>
I20260925 15:16:07.001674 26224 fs_manager.cc:359] Metadata directory not provided
I20260925 15:16:07.001773 26224 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:16:07.002008 26224 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:16:07.004927 26224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-0/data/instance:
uuid: "e5c51475c8dd40cfb94ec868907836ca"
format_stamp: "Formatted at 2026-09-25 15:16:07 on dist-test-slave-p468"
server_key: "02203ba1aa34ee214db4ff6de2e8974d"
server_key_iv: "697a5fac1ab3e5e763f6fb0ef37ec69e"
server_key_version: "encryptionkey@0"
I20260925 15:16:07.005630 26224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-0/wal/instance:
uuid: "e5c51475c8dd40cfb94ec868907836ca"
format_stamp: "Formatted at 2026-09-25 15:16:07 on dist-test-slave-p468"
server_key: "02203ba1aa34ee214db4ff6de2e8974d"
server_key_iv: "697a5fac1ab3e5e763f6fb0ef37ec69e"
server_key_version: "encryptionkey@0"
I20260925 15:16:07.010645 26224 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260925 15:16:07.013996 26239 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:07.015470 26224 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:16:07.015652 26224 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-0/wal
uuid: "e5c51475c8dd40cfb94ec868907836ca"
format_stamp: "Formatted at 2026-09-25 15:16:07 on dist-test-slave-p468"
server_key: "02203ba1aa34ee214db4ff6de2e8974d"
server_key_iv: "697a5fac1ab3e5e763f6fb0ef37ec69e"
server_key_version: "encryptionkey@0"
I20260925 15:16:07.015846 26224 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:16:07.034539 26224 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:16:07.038442 26224 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:16:07.038726 26224 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:16:07.039621 26224 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:16:07.040900 26224 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:16:07.041002 26224 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:07.041162 26224 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:16:07.041237 26224 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:07.074244 26224 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.1:44123
I20260925 15:16:07.074347 26351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.1:44123 every 8 connection(s)
I20260925 15:16:07.075672 26224 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-0/data/info.pb
I20260925 15:16:07.083711 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 26224
I20260925 15:16:07.083933 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-0/wal/instance
I20260925 15:16:07.084585 12320 external_mini_cluster.cc:1428] Setting key 280a118b801ec40b679ed547c8c2bd67
Sep 25 15:16:07 dist-test-slave-p468 krb5kdc[26118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349366, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.1@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
I20260925 15:16:07.094947 26354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:07.078093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.1:48567 (local address 127.12.8.62:33201)
0925 15:16:07.078538 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:07.078543 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:07.079623 (+ 1080us) server_negotiation.cc:407] Connection header received
0925 15:16:07.081383 (+ 1760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:07.081388 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:07.081455 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:07.081553 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:07.083029 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:07.083741 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:07.085399 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:07.085935 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:07.086056 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:07.089420 (+ 3364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:16:07.089426 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:16:07.089429 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:16:07.089459 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:16:07.092014 (+ 2555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:07.092716 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:07.092724 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:07.092732 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:07.092845 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:07.093273 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:07.093278 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:07.093281 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:07.093528 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:16:07.093691 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:07.094359 (+ 668us) server_negotiation.cc:299] Negotiation successful
0925 15:16:07.094574 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":232,"threads_started":1}
I20260925 15:16:07.095824 26355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:07.078699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:33201 (local address 127.12.8.1:48567)
0925 15:16:07.079414 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:07.079451 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:07.081137 (+ 1686us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:07.081750 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:07.081757 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:07.082350 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:16:07.082873 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:07.082889 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:07.083918 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:07.083922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:07.085223 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:07.085235 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:07.086158 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:07.086163 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:07.086477 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:07.087121 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:16:07.087170 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:16:07.089208 (+ 2038us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:16:07.092161 (+ 2953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:16:07.092169 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:16:07.092184 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:16:07.092581 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:16:07.092955 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:16:07.092960 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:16:07.092963 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:16:07.093154 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:16:07.093656 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:16:07.093663 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:16:07.093977 (+ 314us) client_negotiation.cc:769] Sending connection context
0925 15:16:07.094256 (+ 279us) client_negotiation.cc:240] Negotiation successful
0925 15:16:07.094743 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":293,"threads_started":1}
I20260925 15:16:07.097402 26352 heartbeater.cc:347] Connected to a master server at 127.12.8.62:33201
I20260925 15:16:07.097694 26352 heartbeater.cc:464] Registering TS with master...
I20260925 15:16:07.098546 26352 heartbeater.cc:511] Master 127.12.8.62:33201 requested a full tablet report, sending...
I20260925 15:16:07.100801 26163 ts_manager.cc:194] Registered new tserver with Master: e5c51475c8dd40cfb94ec868907836ca (127.12.8.1:44123)
I20260925 15:16:07.103008 26163 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.1@KRBTEST.COM'} at 127.12.8.1:48567
WARNING: no policy specified for kudu/127.12.8.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.2.keytab.
Entry for principal kudu/127.12.8.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.2.keytab.
I20260925 15:16:07.135756 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.2:0
--local_ip_for_outbound_sockets=127.12.8.2
--webserver_interface=127.12.8.2
--webserver_port=0
--tserver_master_addrs=127.12.8.62:33201
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.2.keytab
--principal=kudu/127.12.8.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:35145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:16:07.448069 26358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:16:07.448336 26358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:16:07.448431 26358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:16:07.455535 26358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:16:07.455634 26358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:16:07.455742 26358 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.2
I20260925 15:16:07.464917 26358 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:35145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.2.keytab
--principal=kudu/127.12.8.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.8.2
--webserver_port=0
--tserver_master_addrs=127.12.8.62:33201
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.2
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:16:07.466005 26358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:16:07.467283 26358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:16:07.480019 26363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:16:07.480065 26364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:16:07.480091 26366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:16:07.480899 26358 server_base.cc:1034] running on GCE node
I20260925 15:16:07.481565 26358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:16:07.482628 26358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:16:07.483894 26358 hybrid_clock.cc:648] HybridClock initialized: now 1790349367483845 us; error 64 us; skew 500 ppm
I20260925 15:16:07.484244 26358 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:16:07 dist-test-slave-p468 krb5kdc[26118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349367, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:16:07.487804 26358 init.cc:377] Logged in from keytab as kudu/127.12.8.2@KRBTEST.COM (short username kudu)
I20260925 15:16:07.489365 26358 webserver.cc:460] Webserver started at http://127.12.8.2:34027/ using document root <none> and password file <none>
I20260925 15:16:07.490059 26358 fs_manager.cc:359] Metadata directory not provided
I20260925 15:16:07.490238 26358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:16:07.490563 26358 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:16:07.493520 26358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-1/data/instance:
uuid: "1b034ca722914ac1ac4b23691a36c014"
format_stamp: "Formatted at 2026-09-25 15:16:07 on dist-test-slave-p468"
server_key: "18f60b2de7d1d18d7fc5dca16b0ab3ee"
server_key_iv: "282252730ca62333142d9e7f224eafdc"
server_key_version: "encryptionkey@0"
I20260925 15:16:07.494438 26358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-1/wal/instance:
uuid: "1b034ca722914ac1ac4b23691a36c014"
format_stamp: "Formatted at 2026-09-25 15:16:07 on dist-test-slave-p468"
server_key: "18f60b2de7d1d18d7fc5dca16b0ab3ee"
server_key_iv: "282252730ca62333142d9e7f224eafdc"
server_key_version: "encryptionkey@0"
I20260925 15:16:07.499752 26358 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260925 15:16:07.503324 26373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:07.504776 26358 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 15:16:07.504938 26358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-1/wal
uuid: "1b034ca722914ac1ac4b23691a36c014"
format_stamp: "Formatted at 2026-09-25 15:16:07 on dist-test-slave-p468"
server_key: "18f60b2de7d1d18d7fc5dca16b0ab3ee"
server_key_iv: "282252730ca62333142d9e7f224eafdc"
server_key_version: "encryptionkey@0"
I20260925 15:16:07.505131 26358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:16:07.521533 26358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:16:07.525635 26358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:16:07.525913 26358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:16:07.526856 26358 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:16:07.528122 26358 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:16:07.528191 26358 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:07.528349 26358 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:16:07.528401 26358 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:07.561060 26358 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.2:46647
I20260925 15:16:07.561193 26485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.2:46647 every 8 connection(s)
I20260925 15:16:07.562320 26358 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-1/data/info.pb
I20260925 15:16:07.565315 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 26358
I20260925 15:16:07.565479 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-1/wal/instance
I20260925 15:16:07.565954 12320 external_mini_cluster.cc:1428] Setting key 32dc2107cdfbfba755eff68b412099c4
Sep 25 15:16:07 dist-test-slave-p468 krb5kdc[26118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349367, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.2@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
I20260925 15:16:07.583689 26354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:07.564812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.2:39073 (local address 127.12.8.62:33201)
0925 15:16:07.565061 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:07.565067 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:07.566418 (+ 1351us) server_negotiation.cc:407] Connection header received
0925 15:16:07.568206 (+ 1788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:07.568213 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:07.568305 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:07.568414 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:07.569815 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:07.571055 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:07.572567 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:07.572983 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:07.573121 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:07.576864 (+ 3743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:16:07.576874 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:16:07.576881 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:16:07.576925 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:16:07.580424 (+ 3499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:07.581182 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:07.581190 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:07.581195 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:07.581356 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:07.581748 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:07.581755 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:07.581761 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:07.582148 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:16:07.582337 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:07.582981 (+ 644us) server_negotiation.cc:299] Negotiation successful
0925 15:16:07.583244 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 15:16:07.584265 26488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:07.565340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:33201 (local address 127.12.8.2:39073)
0925 15:16:07.566251 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:07.566300 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:07.567948 (+ 1648us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:07.568601 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:07.568608 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:07.569111 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:16:07.569637 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:07.569665 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:07.571205 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:07.571209 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:07.572412 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:07.572424 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:07.573208 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:07.573216 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:07.573454 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:07.574126 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:16:07.574152 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:16:07.576645 (+ 2493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:16:07.580592 (+ 3947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:16:07.580601 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:16:07.580619 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:16:07.581067 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:16:07.581466 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:16:07.581469 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:16:07.581471 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:16:07.581637 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:16:07.582308 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:16:07.582313 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:16:07.582607 (+ 294us) client_negotiation.cc:769] Sending connection context
0925 15:16:07.582843 (+ 236us) client_negotiation.cc:240] Negotiation successful
0925 15:16:07.583202 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":380,"threads_started":1}
I20260925 15:16:07.585654 26486 heartbeater.cc:347] Connected to a master server at 127.12.8.62:33201
I20260925 15:16:07.585968 26486 heartbeater.cc:464] Registering TS with master...
I20260925 15:16:07.586963 26486 heartbeater.cc:511] Master 127.12.8.62:33201 requested a full tablet report, sending...
I20260925 15:16:07.588657 26163 ts_manager.cc:194] Registered new tserver with Master: 1b034ca722914ac1ac4b23691a36c014 (127.12.8.2:46647)
I20260925 15:16:07.589445 26163 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.2@KRBTEST.COM'} at 127.12.8.2:39073
WARNING: no policy specified for kudu/127.12.8.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.3.keytab.
Entry for principal kudu/127.12.8.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.3.keytab.
I20260925 15:16:07.621446 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.3:0
--local_ip_for_outbound_sockets=127.12.8.3
--webserver_interface=127.12.8.3
--webserver_port=0
--tserver_master_addrs=127.12.8.62:33201
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.3.keytab
--principal=kudu/127.12.8.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:35145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:16:07.956815 26491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:16:07.957125 26491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:16:07.957296 26491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:16:07.964438 26491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:16:07.964577 26491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:16:07.964766 26491 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.3
I20260925 15:16:07.974313 26491 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:35145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.3.keytab
--principal=kudu/127.12.8.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.8.3
--webserver_port=0
--tserver_master_addrs=127.12.8.62:33201
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.3
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:16:07.975569 26491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:16:07.976869 26491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:16:07.990051 26496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:16:07.990139 26499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:16:07.990043 26497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:16:07.991074 26491 server_base.cc:1034] running on GCE node
I20260925 15:16:07.991652 26491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:16:07.992565 26491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:16:07.993758 26491 hybrid_clock.cc:648] HybridClock initialized: now 1790349367993719 us; error 37 us; skew 500 ppm
I20260925 15:16:07.994081 26491 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:16:07 dist-test-slave-p468 krb5kdc[26118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349367, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:16:07.997289 26491 init.cc:377] Logged in from keytab as kudu/127.12.8.3@KRBTEST.COM (short username kudu)
I20260925 15:16:07.998747 26491 webserver.cc:460] Webserver started at http://127.12.8.3:32787/ using document root <none> and password file <none>
I20260925 15:16:07.999413 26491 fs_manager.cc:359] Metadata directory not provided
I20260925 15:16:07.999555 26491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:16:07.999850 26491 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:16:08.002791 26491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-2/data/instance:
uuid: "33c2d12159954b88ae9974464003472f"
format_stamp: "Formatted at 2026-09-25 15:16:07 on dist-test-slave-p468"
server_key: "eb96f366841014226ddfe887ce681d42"
server_key_iv: "24329e77fbeb1b69e7aaa732ed670f0c"
server_key_version: "encryptionkey@0"
I20260925 15:16:08.003548 26491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-2/wal/instance:
uuid: "33c2d12159954b88ae9974464003472f"
format_stamp: "Formatted at 2026-09-25 15:16:07 on dist-test-slave-p468"
server_key: "eb96f366841014226ddfe887ce681d42"
server_key_iv: "24329e77fbeb1b69e7aaa732ed670f0c"
server_key_version: "encryptionkey@0"
I20260925 15:16:08.008498 26491 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 15:16:08.011875 26506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:08.013274 26491 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 15:16:08.013445 26491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-2/wal
uuid: "33c2d12159954b88ae9974464003472f"
format_stamp: "Formatted at 2026-09-25 15:16:07 on dist-test-slave-p468"
server_key: "eb96f366841014226ddfe887ce681d42"
server_key_iv: "24329e77fbeb1b69e7aaa732ed670f0c"
server_key_version: "encryptionkey@0"
I20260925 15:16:08.013650 26491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:16:08.028316 26491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:16:08.032536 26491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:16:08.032833 26491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:16:08.033869 26491 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:16:08.035298 26491 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:16:08.035403 26491 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:08.035564 26491 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:16:08.035640 26491 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:08.068141 26491 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.3:41841
I20260925 15:16:08.068230 26618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.3:41841 every 8 connection(s)
I20260925 15:16:08.069386 26491 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-2/data/info.pb
I20260925 15:16:08.074047 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 26491
I20260925 15:16:08.074263 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-2/wal/instance
I20260925 15:16:08.074826 12320 external_mini_cluster.cc:1428] Setting key c1bcd94cae3a3e0847f5c2ade4423768
Sep 25 15:16:08 dist-test-slave-p468 krb5kdc[26118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349367, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.3@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
I20260925 15:16:08.090152 26354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:08.072022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.3:54251 (local address 127.12.8.62:33201)
0925 15:16:08.072215 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:08.072220 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:08.073297 (+ 1077us) server_negotiation.cc:407] Connection header received
0925 15:16:08.075888 (+ 2591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:08.075896 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:08.075991 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:08.076089 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:08.078051 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:08.078945 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:08.080719 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:08.081097 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:08.081196 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:08.084598 (+ 3402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:16:08.084602 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:16:08.084605 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:16:08.084636 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:16:08.087293 (+ 2657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:08.087907 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:08.087910 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:08.087913 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:08.088065 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:08.088458 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:08.088466 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:08.088469 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:08.088757 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:16:08.088918 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:08.089599 (+ 681us) server_negotiation.cc:299] Negotiation successful
0925 15:16:08.089827 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 15:16:08.091055 26621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:08.072436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:33201 (local address 127.12.8.3:54251)
0925 15:16:08.073134 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:08.073184 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:08.075598 (+ 2414us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:08.076428 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:08.076439 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:08.077125 (+ 686us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:16:08.077874 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:08.077897 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:08.079099 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:08.079104 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:08.080576 (+ 1472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:08.080588 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:08.081318 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:08.081323 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:08.081551 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:08.082263 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:16:08.082285 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:16:08.084413 (+ 2128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:16:08.087415 (+ 3002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:16:08.087421 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:16:08.087431 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:16:08.087808 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:16:08.088173 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:16:08.088177 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:16:08.088179 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:16:08.088340 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:16:08.088887 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:16:08.088894 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:16:08.089223 (+ 329us) client_negotiation.cc:769] Sending connection context
0925 15:16:08.089477 (+ 254us) client_negotiation.cc:240] Negotiation successful
0925 15:16:08.089959 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":282,"threads_started":1}
I20260925 15:16:08.092320 26619 heartbeater.cc:347] Connected to a master server at 127.12.8.62:33201
I20260925 15:16:08.092604 26619 heartbeater.cc:464] Registering TS with master...
I20260925 15:16:08.093291 26619 heartbeater.cc:511] Master 127.12.8.62:33201 requested a full tablet report, sending...
I20260925 15:16:08.094657 26163 ts_manager.cc:194] Registered new tserver with Master: 33c2d12159954b88ae9974464003472f (127.12.8.3:41841)
I20260925 15:16:08.095439 26163 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.3@KRBTEST.COM'} at 127.12.8.3:54251
I20260925 15:16:08.102113 12320 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:16:08.106559 26352 heartbeater.cc:503] Master 127.12.8.62:33201 was elected leader, sending a full tablet report...
I20260925 15:16:08.119167 26354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:08.106800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53362 (local address 127.12.8.62:33201)
0925 15:16:08.106991 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:08.106995 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:08.107516 (+ 521us) server_negotiation.cc:407] Connection header received
0925 15:16:08.107712 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:08.107717 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:08.107807 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:08.107902 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:08.109294 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:08.110080 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:08.111386 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:08.111779 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:08.111873 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:08.114124 (+ 2251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:16:08.114129 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:16:08.114132 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:16:08.114172 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:16:08.116585 (+ 2413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:08.117305 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:08.117309 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:08.117311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:08.117433 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:08.117746 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:08.117749 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:08.117751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:08.118103 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:16:08.118358 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:08.118688 (+ 330us) server_negotiation.cc:299] Negotiation successful
0925 15:16:08.118849 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 15:16:08.122973 26163 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53362:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:16:08.129792 26163 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:16:08.164148 26631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:08.141206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.2:46647 (local address 127.0.0.1:54270)
0925 15:16:08.153381 (+ 12175us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:08.153429 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:08.153597 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:08.156653 (+ 3056us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:08.156660 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:08.156691 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:16:08.157090 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:08.157101 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:08.159233 (+ 2132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:08.159239 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:08.161323 (+ 2084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:08.161335 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:08.163062 (+ 1727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:08.163069 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:08.163260 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:08.163315 (+ 55us) client_negotiation.cc:769] Sending connection context
0925 15:16:08.163444 (+ 129us) client_negotiation.cc:240] Negotiation successful
0925 15:16:08.163637 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11908,"mutex_wait_us":120,"thread_start_us":12559,"threads_started":1}
I20260925 15:16:08.164639 26633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:08.141475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.3:41841 (local address 127.0.0.1:49364)
0925 15:16:08.155398 (+ 13923us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:08.155412 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:08.155504 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:08.156350 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:08.156358 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:08.156385 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:16:08.156839 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:08.156853 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:08.159557 (+ 2704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:08.159563 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:08.161777 (+ 2214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:08.161791 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:08.163529 (+ 1738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:08.163535 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:08.163706 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:08.163737 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 15:16:08.163827 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 15:16:08.163954 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13839,"mutex_wait_us":49,"thread_start_us":14132,"threads_started":1}
I20260925 15:16:08.165016 26632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:08.141353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54270 (local address 127.12.8.2:46647)
0925 15:16:08.156051 (+ 14698us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:08.156060 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:08.156086 (+ 26us) server_negotiation.cc:407] Connection header received
0925 15:16:08.156192 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:08.156200 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:08.156433 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:08.157327 (+ 894us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:16:08.157538 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:08.159080 (+ 1542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:08.161484 (+ 2404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:08.162869 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:08.163001 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:08.163244 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:08.164501 (+ 1257us) server_negotiation.cc:299] Negotiation successful
0925 15:16:08.164691 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14521,"thread_start_us":15021,"threads_started":1}
I20260925 15:16:08.165243 26634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:08.143656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49364 (local address 127.12.8.3:41841)
0925 15:16:08.152480 (+ 8824us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:08.152491 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:08.155499 (+ 3008us) server_negotiation.cc:407] Connection header received
0925 15:16:08.155664 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:08.155685 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:08.155946 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:08.156155 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:16:08.157076 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:08.159427 (+ 2351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:08.161935 (+ 2508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:08.163328 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:08.163469 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:08.164686 (+ 1217us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:08.164760 (+ 74us) server_negotiation.cc:299] Negotiation successful
0925 15:16:08.164911 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8615,"thread_start_us":8968,"threads_started":1}
I20260925 15:16:08.165354 26635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:08.156153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.1:44123 (local address 127.0.0.1:60162)
0925 15:16:08.157388 (+ 1235us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:08.157401 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:08.157541 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:08.158070 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:08.158074 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:08.158098 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:16:08.158575 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:08.158588 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:08.160312 (+ 1724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:08.160317 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:08.162377 (+ 2060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:08.162388 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:08.164275 (+ 1887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:08.164284 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:08.164730 (+ 446us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:08.164762 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 15:16:08.164853 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 15:16:08.164965 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1145,"mutex_wait_us":3,"thread_start_us":1510,"threads_started":1}
I20260925 15:16:08.165762 26636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:08.156237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60162 (local address 127.12.8.1:44123)
0925 15:16:08.157068 (+ 831us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:08.157074 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:08.157502 (+ 428us) server_negotiation.cc:407] Connection header received
0925 15:16:08.157700 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:08.157708 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:08.157899 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:08.158080 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:16:08.158799 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:08.160152 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:08.162523 (+ 2371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:08.163774 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:08.163896 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:08.164113 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:08.165262 (+ 1149us) server_negotiation.cc:299] Negotiation successful
0925 15:16:08.165442 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":702,"thread_start_us":546,"threads_started":1}
I20260925 15:16:08.168708 26287 tablet_service.cc:1467] Processing CreateTablet for tablet e1608fe1f4b841dbb902fff101a36b68 (DEFAULT_TABLE table=test-table [id=57869b5eac79460e905bbe1700d5dcd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:16:08.168814 26421 tablet_service.cc:1467] Processing CreateTablet for tablet e1608fe1f4b841dbb902fff101a36b68 (DEFAULT_TABLE table=test-table [id=57869b5eac79460e905bbe1700d5dcd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:16:08.169348 26554 tablet_service.cc:1467] Processing CreateTablet for tablet e1608fe1f4b841dbb902fff101a36b68 (DEFAULT_TABLE table=test-table [id=57869b5eac79460e905bbe1700d5dcd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:16:08.169951 26287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1608fe1f4b841dbb902fff101a36b68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:16:08.170126 26421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1608fe1f4b841dbb902fff101a36b68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:16:08.170604 26554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1608fe1f4b841dbb902fff101a36b68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:16:08.180903 26637 tablet_bootstrap.cc:492] T e1608fe1f4b841dbb902fff101a36b68 P e5c51475c8dd40cfb94ec868907836ca: Bootstrap starting.
I20260925 15:16:08.184456 26637 tablet_bootstrap.cc:654] T e1608fe1f4b841dbb902fff101a36b68 P e5c51475c8dd40cfb94ec868907836ca: Neither blocks nor log segments found. Creating new log.
I20260925 15:16:08.184968 26639 tablet_bootstrap.cc:492] T e1608fe1f4b841dbb902fff101a36b68 P 33c2d12159954b88ae9974464003472f: Bootstrap starting.
I20260925 15:16:08.185860 26637 log.cc:824] T e1608fe1f4b841dbb902fff101a36b68 P e5c51475c8dd40cfb94ec868907836ca: Log is configured to *not* fsync() on all Append() calls
I20260925 15:16:08.186641 26638 tablet_bootstrap.cc:492] T e1608fe1f4b841dbb902fff101a36b68 P 1b034ca722914ac1ac4b23691a36c014: Bootstrap starting.
I20260925 15:16:08.189257 26638 tablet_bootstrap.cc:654] T e1608fe1f4b841dbb902fff101a36b68 P 1b034ca722914ac1ac4b23691a36c014: Neither blocks nor log segments found. Creating new log.
I20260925 15:16:08.189355 26639 tablet_bootstrap.cc:654] T e1608fe1f4b841dbb902fff101a36b68 P 33c2d12159954b88ae9974464003472f: Neither blocks nor log segments found. Creating new log.
I20260925 15:16:08.190558 26638 log.cc:824] T e1608fe1f4b841dbb902fff101a36b68 P 1b034ca722914ac1ac4b23691a36c014: Log is configured to *not* fsync() on all Append() calls
I20260925 15:16:08.190658 26639 log.cc:824] T e1608fe1f4b841dbb902fff101a36b68 P 33c2d12159954b88ae9974464003472f: Log is configured to *not* fsync() on all Append() calls
I20260925 15:16:08.192641 26637 tablet_bootstrap.cc:492] T e1608fe1f4b841dbb902fff101a36b68 P e5c51475c8dd40cfb94ec868907836ca: No bootstrap required, opened a new log
I20260925 15:16:08.192910 26637 ts_tablet_manager.cc:1397] T e1608fe1f4b841dbb902fff101a36b68 P e5c51475c8dd40cfb94ec868907836ca: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.004s
I20260925 15:16:08.193480 26638 tablet_bootstrap.cc:492] T e1608fe1f4b841dbb902fff101a36b68 P 1b034ca722914ac1ac4b23691a36c014: No bootstrap required, opened a new log
I20260925 15:16:08.193686 26639 tablet_bootstrap.cc:492] T e1608fe1f4b841dbb902fff101a36b68 P 33c2d12159954b88ae9974464003472f: No bootstrap required, opened a new log
I20260925 15:16:08.193746 26638 ts_tablet_manager.cc:1397] T e1608fe1f4b841dbb902fff101a36b68 P 1b034ca722914ac1ac4b23691a36c014: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260925 15:16:08.193928 26639 ts_tablet_manager.cc:1397] T e1608fe1f4b841dbb902fff101a36b68 P 33c2d12159954b88ae9974464003472f: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260925 15:16:08.200109 26637 raft_consensus.cc:348] T e1608fe1f4b841dbb902fff101a36b68 P e5c51475c8dd40cfb94ec868907836ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b034ca722914ac1ac4b23691a36c014" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 46647 } } peers { permanent_uuid: "33c2d12159954b88ae9974464003472f" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 41841 } } peers { permanent_uuid: "e5c51475c8dd40cfb94ec868907836ca" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 44123 } }
I20260925 15:16:08.200479 26637 raft_consensus.cc:374] T e1608fe1f4b841dbb902fff101a36b68 P e5c51475c8dd40cfb94ec868907836ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:16:08.200600 26637 raft_consensus.cc:729] T e1608fe1f4b841dbb902fff101a36b68 P e5c51475c8dd40cfb94ec868907836ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5c51475c8dd40cfb94ec868907836ca, State: Initialized, Role: FOLLOWER
I20260925 15:16:08.201187 26637 consensus_queue.cc:260] T e1608fe1f4b841dbb902fff101a36b68 P e5c51475c8dd40cfb94ec868907836ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b034ca722914ac1ac4b23691a36c014" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 46647 } } peers { permanent_uuid: "33c2d12159954b88ae9974464003472f" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 41841 } } peers { permanent_uuid: "e5c51475c8dd40cfb94ec868907836ca" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 44123 } }
I20260925 15:16:08.202960 26637 ts_tablet_manager.cc:1428] T e1608fe1f4b841dbb902fff101a36b68 P e5c51475c8dd40cfb94ec868907836ca: Time spent starting tablet: real 0.010s user 0.011s sys 0.001s
I20260925 15:16:08.204166 26638 raft_consensus.cc:348] T e1608fe1f4b841dbb902fff101a36b68 P 1b034ca722914ac1ac4b23691a36c014 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b034ca722914ac1ac4b23691a36c014" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 46647 } } peers { permanent_uuid: "33c2d12159954b88ae9974464003472f" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 41841 } } peers { permanent_uuid: "e5c51475c8dd40cfb94ec868907836ca" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 44123 } }
I20260925 15:16:08.204633 26638 raft_consensus.cc:374] T e1608fe1f4b841dbb902fff101a36b68 P 1b034ca722914ac1ac4b23691a36c014 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:16:08.204746 26638 raft_consensus.cc:729] T e1608fe1f4b841dbb902fff101a36b68 P 1b034ca722914ac1ac4b23691a36c014 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b034ca722914ac1ac4b23691a36c014, State: Initialized, Role: FOLLOWER
I20260925 15:16:08.204996 26639 raft_consensus.cc:348] T e1608fe1f4b841dbb902fff101a36b68 P 33c2d12159954b88ae9974464003472f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b034ca722914ac1ac4b23691a36c014" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 46647 } } peers { permanent_uuid: "33c2d12159954b88ae9974464003472f" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 41841 } } peers { permanent_uuid: "e5c51475c8dd40cfb94ec868907836ca" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 44123 } }
I20260925 15:16:08.205327 26639 raft_consensus.cc:374] T e1608fe1f4b841dbb902fff101a36b68 P 33c2d12159954b88ae9974464003472f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:16:08.205438 26639 raft_consensus.cc:729] T e1608fe1f4b841dbb902fff101a36b68 P 33c2d12159954b88ae9974464003472f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33c2d12159954b88ae9974464003472f, State: Initialized, Role: FOLLOWER
I20260925 15:16:08.205367 26638 consensus_queue.cc:260] T e1608fe1f4b841dbb902fff101a36b68 P 1b034ca722914ac1ac4b23691a36c014 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b034ca722914ac1ac4b23691a36c014" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 46647 } } peers { permanent_uuid: "33c2d12159954b88ae9974464003472f" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 41841 } } peers { permanent_uuid: "e5c51475c8dd40cfb94ec868907836ca" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 44123 } }
I20260925 15:16:08.206039 26639 consensus_queue.cc:260] T e1608fe1f4b841dbb902fff101a36b68 P 33c2d12159954b88ae9974464003472f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b034ca722914ac1ac4b23691a36c014" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 46647 } } peers { permanent_uuid: "33c2d12159954b88ae9974464003472f" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 41841 } } peers { permanent_uuid: "e5c51475c8dd40cfb94ec868907836ca" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 44123 } }
I20260925 15:16:08.206503 26486 heartbeater.cc:503] Master 127.12.8.62:33201 was elected leader, sending a full tablet report...
I20260925 15:16:08.207146 26638 ts_tablet_manager.cc:1428] T e1608fe1f4b841dbb902fff101a36b68 P 1b034ca722914ac1ac4b23691a36c014: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260925 15:16:08.207975 26619 heartbeater.cc:503] Master 127.12.8.62:33201 was elected leader, sending a full tablet report...
I20260925 15:16:08.208981 26639 ts_tablet_manager.cc:1428] T e1608fe1f4b841dbb902fff101a36b68 P 33c2d12159954b88ae9974464003472f: Time spent starting tablet: real 0.015s user 0.007s sys 0.007s
I20260925 15:16:08.284644 26644 raft_consensus.cc:482] T e1608fe1f4b841dbb902fff101a36b68 P 1b034ca722914ac1ac4b23691a36c014 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:16:08.284974 26644 raft_consensus.cc:504] T e1608fe1f4b841dbb902fff101a36b68 P 1b034ca722914ac1ac4b23691a36c014 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b034ca722914ac1ac4b23691a36c014" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 46647 } } peers { permanent_uuid: "33c2d12159954b88ae9974464003472f" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 41841 } } peers { permanent_uuid: "e5c51475c8dd40cfb94ec868907836ca" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 44123 } }
I20260925 15:16:08.286756 26644 leader_election.cc:288] T e1608fe1f4b841dbb902fff101a36b68 P 1b034ca722914ac1ac4b23691a36c014 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 33c2d12159954b88ae9974464003472f (127.12.8.3:41841), e5c51475c8dd40cfb94ec868907836ca (127.12.8.1:44123)
I20260925 15:16:08.294844 26634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:08.287287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.2:47009 (local address 127.12.8.3:41841)
0925 15:16:08.287481 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:08.287485 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:08.287982 (+ 497us) server_negotiation.cc:407] Connection header received
0925 15:16:08.288217 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:08.288224 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:08.288301 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:08.288404 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:16:08.289313 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:08.290372 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:08.292936 (+ 2564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:08.293917 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:08.294067 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:08.294174 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:08.294470 (+ 296us) server_negotiation.cc:299] Negotiation successful
0925 15:16:08.294590 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 15:16:08.295027 26646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:08.287147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.3:41841 (local address 127.12.8.2:47009)
0925 15:16:08.287861 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:08.287884 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:08.288037 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:08.288590 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:08.288596 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:08.288624 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:16:08.289166 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:08.289178 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:08.290515 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:08.290527 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:08.292740 (+ 2213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:08.292754 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:08.294125 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:08.294131 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:08.294315 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:08.294338 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 15:16:08.294447 (+ 109us) client_negotiation.cc:240] Negotiation successful
0925 15:16:08.294577 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"spinlock_wait_cycles":36096,"thread_start_us":356,"threads_started":1}
I20260925 15:16:08.295388 26636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:08.288303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.2:50861 (local address 127.12.8.1:44123)
0925 15:16:08.288504 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:08.288510 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:08.288772 (+ 262us) server_negotiation.cc:407] Connection header received
0925 15:16:08.288957 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:08.288960 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:08.289017 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:08.289116 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:16:08.289700 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:08.290694 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:08.293542 (+ 2848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:08.294369 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:08.294532 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:08.294638 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:08.294953 (+ 315us) server_negotiation.cc:299] Negotiation successful
0925 15:16:08.295096 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 15:16:08.295424 26647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:08.287983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.1:44123 (local address 127.12.8.2:50861)
0925 15:16:08.288662 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:08.288687 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:08.288835 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:08.289171 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:08.289176 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:08.289203 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:16:08.289527 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:08.289535 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:08.290838 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:08.290843 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:08.293333 (+ 2490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:08.293348 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:08.294606 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:08.294613 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:08.294798 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:08.294821 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 15:16:08.294928 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 15:16:08.295077 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":367,"threads_started":1}
I20260925 15:16:08.296147 26574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1608fe1f4b841dbb902fff101a36b68" candidate_uuid: "1b034ca722914ac1ac4b23691a36c014" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33c2d12159954b88ae9974464003472f" is_pre_election: true
I20260925 15:16:08.296586 26574 raft_consensus.cc:2454] T e1608fe1f4b841dbb902fff101a36b68 P 33c2d12159954b88ae9974464003472f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b034ca722914ac1ac4b23691a36c014 in term 0.
I20260925 15:16:08.297258 26374 leader_election.cc:302] T e1608fe1f4b841dbb902fff101a36b68 P 1b034ca722914ac1ac4b23691a36c014 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b034ca722914ac1ac4b23691a36c014, 33c2d12159954b88ae9974464003472f; no voters:
I20260925 15:16:08.297286 26307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1608fe1f4b841dbb902fff101a36b68" candidate_uuid: "1b034ca722914ac1ac4b23691a36c014" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5c51475c8dd40cfb94ec868907836ca" is_pre_election: true
I20260925 15:16:08.297642 26644 raft_consensus.cc:2781] T e1608fe1f4b841dbb902fff101a36b68 P 1b034ca722914ac1ac4b23691a36c014 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:16:08.297793 26644 raft_consensus.cc:482] T e1608fe1f4b841dbb902fff101a36b68 P 1b034ca722914ac1ac4b23691a36c014 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:16:08.297912 26644 raft_consensus.cc:3037] T e1608fe1f4b841dbb902fff101a36b68 P 1b034ca722914ac1ac4b23691a36c014 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:16:08.297825 26307 raft_consensus.cc:2454] T e1608fe1f4b841dbb902fff101a36b68 P e5c51475c8dd40cfb94ec868907836ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b034ca722914ac1ac4b23691a36c014 in term 0.
I20260925 15:16:08.300206 26644 raft_consensus.cc:504] T e1608fe1f4b841dbb902fff101a36b68 P 1b034ca722914ac1ac4b23691a36c014 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b034ca722914ac1ac4b23691a36c014" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 46647 } } peers { permanent_uuid: "33c2d12159954b88ae9974464003472f" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 41841 } } peers { permanent_uuid: "e5c51475c8dd40cfb94ec868907836ca" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 44123 } }
I20260925 15:16:08.300725 26644 leader_election.cc:288] T e1608fe1f4b841dbb902fff101a36b68 P 1b034ca722914ac1ac4b23691a36c014 [CANDIDATE]: Term 1 election: Requested vote from peers 33c2d12159954b88ae9974464003472f (127.12.8.3:41841), e5c51475c8dd40cfb94ec868907836ca (127.12.8.1:44123)
I20260925 15:16:08.301293 26574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1608fe1f4b841dbb902fff101a36b68" candidate_uuid: "1b034ca722914ac1ac4b23691a36c014" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33c2d12159954b88ae9974464003472f"
I20260925 15:16:08.301354 26307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1608fe1f4b841dbb902fff101a36b68" candidate_uuid: "1b034ca722914ac1ac4b23691a36c014" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5c51475c8dd40cfb94ec868907836ca"
I20260925 15:16:08.301519 26574 raft_consensus.cc:3037] T e1608fe1f4b841dbb902fff101a36b68 P 33c2d12159954b88ae9974464003472f [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:16:08.301537 26307 raft_consensus.cc:3037] T e1608fe1f4b841dbb902fff101a36b68 P e5c51475c8dd40cfb94ec868907836ca [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:16:08.304764 26307 raft_consensus.cc:2454] T e1608fe1f4b841dbb902fff101a36b68 P e5c51475c8dd40cfb94ec868907836ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b034ca722914ac1ac4b23691a36c014 in term 1.
I20260925 15:16:08.304764 26574 raft_consensus.cc:2454] T e1608fe1f4b841dbb902fff101a36b68 P 33c2d12159954b88ae9974464003472f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b034ca722914ac1ac4b23691a36c014 in term 1.
I20260925 15:16:08.305205 26374 leader_election.cc:302] T e1608fe1f4b841dbb902fff101a36b68 P 1b034ca722914ac1ac4b23691a36c014 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b034ca722914ac1ac4b23691a36c014, 33c2d12159954b88ae9974464003472f; no voters:
I20260925 15:16:08.305508 26644 raft_consensus.cc:2781] T e1608fe1f4b841dbb902fff101a36b68 P 1b034ca722914ac1ac4b23691a36c014 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:16:08.305914 26644 raft_consensus.cc:686] T e1608fe1f4b841dbb902fff101a36b68 P 1b034ca722914ac1ac4b23691a36c014 [term 1 LEADER]: Becoming Leader. State: Replica: 1b034ca722914ac1ac4b23691a36c014, State: Running, Role: LEADER
I20260925 15:16:08.306509 26644 consensus_queue.cc:237] T e1608fe1f4b841dbb902fff101a36b68 P 1b034ca722914ac1ac4b23691a36c014 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b034ca722914ac1ac4b23691a36c014" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 46647 } } peers { permanent_uuid: "33c2d12159954b88ae9974464003472f" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 41841 } } peers { permanent_uuid: "e5c51475c8dd40cfb94ec868907836ca" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 44123 } }
I20260925 15:16:08.310850 26163 catalog_manager.cc:5495] T e1608fe1f4b841dbb902fff101a36b68 P 1b034ca722914ac1ac4b23691a36c014 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b034ca722914ac1ac4b23691a36c014 (127.12.8.2). New cstate: current_term: 1 leader_uuid: "1b034ca722914ac1ac4b23691a36c014" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b034ca722914ac1ac4b23691a36c014" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 46647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "33c2d12159954b88ae9974464003472f" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 41841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5c51475c8dd40cfb94ec868907836ca" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 44123 } health_report { overall_health: UNKNOWN } } }
W20260925 15:16:08.314276 26487 tablet.cc:2369] T e1608fe1f4b841dbb902fff101a36b68 P 1b034ca722914ac1ac4b23691a36c014: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:16:08.320966 26620 tablet.cc:2369] T e1608fe1f4b841dbb902fff101a36b68 P 33c2d12159954b88ae9974464003472f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:16:08.328069 26353 tablet.cc:2369] T e1608fe1f4b841dbb902fff101a36b68 P e5c51475c8dd40cfb94ec868907836ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:16:08.342310 26632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:08.336899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54282 (local address 127.12.8.2:46647)
0925 15:16:08.337131 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:08.337136 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:08.337157 (+ 21us) server_negotiation.cc:407] Connection header received
0925 15:16:08.337400 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:08.337404 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:08.337489 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:08.337643 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:16:08.338011 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:08.338753 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:08.340175 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:08.340581 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:08.340724 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:08.341190 (+ 466us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:16:08.341664 (+ 474us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:16:08.341833 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:08.341894 (+ 61us) server_negotiation.cc:299] Negotiation successful
0925 15:16:08.341972 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 15:16:08.350821 26307 raft_consensus.cc:1261] T e1608fe1f4b841dbb902fff101a36b68 P e5c51475c8dd40cfb94ec868907836ca [term 1 FOLLOWER]: Refusing update from remote peer 1b034ca722914ac1ac4b23691a36c014: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:16:08.350837 26574 raft_consensus.cc:1261] T e1608fe1f4b841dbb902fff101a36b68 P 33c2d12159954b88ae9974464003472f [term 1 FOLLOWER]: Refusing update from remote peer 1b034ca722914ac1ac4b23691a36c014: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:16:08.351753 26648 consensus_queue.cc:1035] T e1608fe1f4b841dbb902fff101a36b68 P 1b034ca722914ac1ac4b23691a36c014 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33c2d12159954b88ae9974464003472f" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 41841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:16:08.352170 26644 consensus_queue.cc:1035] T e1608fe1f4b841dbb902fff101a36b68 P 1b034ca722914ac1ac4b23691a36c014 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5c51475c8dd40cfb94ec868907836ca" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 44123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:16:08.377317 26163 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53362:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:16:08.377614 26163 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53362:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:16:08.381319 26163 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 594bdcb833214f8ca0ecc6906c3c8e64: Sending DeleteTablet for 3 replicas of tablet e1608fe1f4b841dbb902fff101a36b68
I20260925 15:16:08.382763 26421 tablet_service.cc:1514] Processing DeleteTablet for tablet e1608fe1f4b841dbb902fff101a36b68 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:16:08 UTC) from {username='kudu', principal='kudu/127.12.8.62@KRBTEST.COM'} at 127.0.0.1:54270
I20260925 15:16:08.382869 26554 tablet_service.cc:1514] Processing DeleteTablet for tablet e1608fe1f4b841dbb902fff101a36b68 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:16:08 UTC) from {username='kudu', principal='kudu/127.12.8.62@KRBTEST.COM'} at 127.0.0.1:49364
I20260925 15:16:08.382843 26287 tablet_service.cc:1514] Processing DeleteTablet for tablet e1608fe1f4b841dbb902fff101a36b68 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:16:08 UTC) from {username='kudu', principal='kudu/127.12.8.62@KRBTEST.COM'} at 127.0.0.1:60162
I20260925 15:16:08.383667 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 26224
I20260925 15:16:08.394482 26663 tablet_replica.cc:330] T e1608fe1f4b841dbb902fff101a36b68 P e5c51475c8dd40cfb94ec868907836ca: stopping tablet replica
I20260925 15:16:08.395202 26663 raft_consensus.cc:2229] T e1608fe1f4b841dbb902fff101a36b68 P e5c51475c8dd40cfb94ec868907836ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:16:08.395944 26663 raft_consensus.cc:2258] T e1608fe1f4b841dbb902fff101a36b68 P e5c51475c8dd40cfb94ec868907836ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:16:08.396579 26661 tablet_replica.cc:330] T e1608fe1f4b841dbb902fff101a36b68 P 1b034ca722914ac1ac4b23691a36c014: stopping tablet replica
I20260925 15:16:08.397188 26662 tablet_replica.cc:330] T e1608fe1f4b841dbb902fff101a36b68 P 33c2d12159954b88ae9974464003472f: stopping tablet replica
I20260925 15:16:08.397629 26662 raft_consensus.cc:2229] T e1608fe1f4b841dbb902fff101a36b68 P 33c2d12159954b88ae9974464003472f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:16:08.398111 26661 raft_consensus.cc:2229] T e1608fe1f4b841dbb902fff101a36b68 P 1b034ca722914ac1ac4b23691a36c014 [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:16:08.398684 26662 raft_consensus.cc:2258] T e1608fe1f4b841dbb902fff101a36b68 P 33c2d12159954b88ae9974464003472f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:16:08.399324 26661 raft_consensus.cc:2258] T e1608fe1f4b841dbb902fff101a36b68 P 1b034ca722914ac1ac4b23691a36c014 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:16:08.399394 26663 ts_tablet_manager.cc:1905] T e1608fe1f4b841dbb902fff101a36b68 P e5c51475c8dd40cfb94ec868907836ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:16:08.401481 26662 ts_tablet_manager.cc:1905] T e1608fe1f4b841dbb902fff101a36b68 P 33c2d12159954b88ae9974464003472f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:16:08.403287 26661 ts_tablet_manager.cc:1905] T e1608fe1f4b841dbb902fff101a36b68 P 1b034ca722914ac1ac4b23691a36c014: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:16:08.406961 26663 ts_tablet_manager.cc:1918] T e1608fe1f4b841dbb902fff101a36b68 P e5c51475c8dd40cfb94ec868907836ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:16:08.407174 26663 log.cc:1196] T e1608fe1f4b841dbb902fff101a36b68 P e5c51475c8dd40cfb94ec868907836ca: Deleting WAL directory at /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-0/wal/wals/e1608fe1f4b841dbb902fff101a36b68
Sep 25 15:16:08 dist-test-slave-p468 krb5kdc[26118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.1@KRBTEST.COM
I20260925 15:16:08.407738 26663 ts_tablet_manager.cc:1939] T e1608fe1f4b841dbb902fff101a36b68 P e5c51475c8dd40cfb94ec868907836ca: Deleting consensus metadata
I20260925 15:16:08.409970 26147 catalog_manager.cc:4841] TS e5c51475c8dd40cfb94ec868907836ca (127.12.8.1:44123): tablet e1608fe1f4b841dbb902fff101a36b68 (table test-table [id=57869b5eac79460e905bbe1700d5dcd8]) successfully deleted
I20260925 15:16:08.413318 26661 ts_tablet_manager.cc:1918] T e1608fe1f4b841dbb902fff101a36b68 P 1b034ca722914ac1ac4b23691a36c014: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:16:08.413580 26661 log.cc:1196] T e1608fe1f4b841dbb902fff101a36b68 P 1b034ca722914ac1ac4b23691a36c014: Deleting WAL directory at /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-1/wal/wals/e1608fe1f4b841dbb902fff101a36b68
I20260925 15:16:08.414274 26661 ts_tablet_manager.cc:1939] T e1608fe1f4b841dbb902fff101a36b68 P 1b034ca722914ac1ac4b23691a36c014: Deleting consensus metadata
I20260925 15:16:08.416178 26148 catalog_manager.cc:4841] TS 1b034ca722914ac1ac4b23691a36c014 (127.12.8.2:46647): tablet e1608fe1f4b841dbb902fff101a36b68 (table test-table [id=57869b5eac79460e905bbe1700d5dcd8]) successfully deleted
I20260925 15:16:08.416827 26636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:08.385072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60176 (local address 127.12.8.1:44123)
0925 15:16:08.385323 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:08.385332 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:08.396847 (+ 11515us) server_negotiation.cc:407] Connection header received
0925 15:16:08.397138 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:08.397144 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:08.397228 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:08.397311 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:08.399322 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:08.400540 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:08.402664 (+ 2124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:08.403302 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:08.403404 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:08.409117 (+ 5713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:16:08.409149 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:16:08.409167 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:16:08.409215 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:16:08.413136 (+ 3921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:08.414095 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:08.414105 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:08.414109 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:08.414305 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:08.414785 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:08.414790 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:08.414793 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:08.415237 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:16:08.415517 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:08.416116 (+ 599us) server_negotiation.cc:299] Negotiation successful
0925 15:16:08.416341 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260925 15:16:08.416833 26662 ts_tablet_manager.cc:1918] T e1608fe1f4b841dbb902fff101a36b68 P 33c2d12159954b88ae9974464003472f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:16:08.417033 26662 log.cc:1196] T e1608fe1f4b841dbb902fff101a36b68 P 33c2d12159954b88ae9974464003472f: Deleting WAL directory at /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-2/wal/wals/e1608fe1f4b841dbb902fff101a36b68
I20260925 15:16:08.417680 26662 ts_tablet_manager.cc:1939] T e1608fe1f4b841dbb902fff101a36b68 P 33c2d12159954b88ae9974464003472f: Deleting consensus metadata
I20260925 15:16:08.417956 26347 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:16:08.419863 26147 catalog_manager.cc:4841] TS 33c2d12159954b88ae9974464003472f (127.12.8.3:41841): tablet e1608fe1f4b841dbb902fff101a36b68 (table test-table [id=57869b5eac79460e905bbe1700d5dcd8]) successfully deleted
I20260925 15:16:08.521864 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 26224
I20260925 15:16:08.547308 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 26358
Sep 25 15:16:08 dist-test-slave-p468 krb5kdc[26118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.2@KRBTEST.COM
I20260925 15:16:08.565584 26632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:08.548273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54288 (local address 127.12.8.2:46647)
0925 15:16:08.548501 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:08.548506 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:08.548578 (+ 72us) server_negotiation.cc:407] Connection header received
0925 15:16:08.548822 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:08.548826 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:08.548892 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:08.548982 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:08.550755 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:08.551682 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:08.553609 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:08.554053 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:08.554151 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:08.559319 (+ 5168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:16:08.559332 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:16:08.559348 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:16:08.559393 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:16:08.562614 (+ 3221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:08.563318 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:08.563331 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:08.563333 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:08.563500 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:08.563912 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:08.563918 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:08.563922 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:08.564265 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:16:08.564617 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:08.564985 (+ 368us) server_negotiation.cc:299] Negotiation successful
0925 15:16:08.565212 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 15:16:08.566329 26481 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:16:08.681663 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 26358
I20260925 15:16:08.699590 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 26491
Sep 25 15:16:08 dist-test-slave-p468 krb5kdc[26118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.3@KRBTEST.COM
I20260925 15:16:08.715270 26634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:08.700391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49378 (local address 127.12.8.3:41841)
0925 15:16:08.700574 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:08.700578 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:08.700632 (+ 54us) server_negotiation.cc:407] Connection header received
0925 15:16:08.700793 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:08.700798 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:08.700869 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:08.700977 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:08.702293 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:08.703081 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:08.704473 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:08.704981 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:08.705086 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:08.709673 (+ 4587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:16:08.709686 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:16:08.709700 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:16:08.709733 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:16:08.712357 (+ 2624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:08.713019 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:08.713027 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:08.713029 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:08.713163 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:08.713530 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:08.713536 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:08.713539 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:08.713884 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:16:08.714192 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:08.714657 (+ 465us) server_negotiation.cc:299] Negotiation successful
0925 15:16:08.714883 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 15:16:08.715911 26614 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:16:08.805965 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 26491
I20260925 15:16:08.822525 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 26131
I20260925 15:16:08.823751 26193 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:16:08.934811 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 26131
I20260925 15:16:08.955873 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.62:33201
--webserver_interface=127.12.8.62
--webserver_port=46131
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab
--principal=kudu/127.12.8.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:35145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.62:33201
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:16:09.271584 26669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:16:09.271859 26669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:16:09.272025 26669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:16:09.279099 26669 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:16:09.279191 26669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:16:09.279307 26669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:16:09.279414 26669 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:16:09.279476 26669 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:16:09.289014 26669 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:35145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.12.8.62:33201
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab
--principal=kudu/127.12.8.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.62:33201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.8.62
--webserver_port=46131
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:16:09.290277 26669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:16:09.291560 26669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:16:09.313832 26678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:16:09.313942 26675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:16:09.313884 26676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:16:09.314978 26669 server_base.cc:1034] running on GCE node
I20260925 15:16:09.315973 26669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:16:09.317582 26669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:16:09.318813 26669 hybrid_clock.cc:648] HybridClock initialized: now 1790349369318744 us; error 68 us; skew 500 ppm
I20260925 15:16:09.319232 26669 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:16:09 dist-test-slave-p468 krb5kdc[26118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349369, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:16:09.323086 26669 init.cc:377] Logged in from keytab as kudu/127.12.8.62@KRBTEST.COM (short username kudu)
I20260925 15:16:09.326036 26669 webserver.cc:460] Webserver started at http://127.12.8.62:46131/ using document root <none> and password file <none>
I20260925 15:16:09.327126 26669 fs_manager.cc:359] Metadata directory not provided
I20260925 15:16:09.327318 26669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:16:09.335067 26669 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 15:16:09.338040 26685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:09.339522 26669 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 15:16:09.339705 26669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/master-0/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/master-0/wal
uuid: "594bdcb833214f8ca0ecc6906c3c8e64"
format_stamp: "Formatted at 2026-09-25 15:16:06 on dist-test-slave-p468"
server_key: "21b8a2d0a9e431406afae1549ce7fdac"
server_key_iv: "2d27c4eb28a0816f2bd06cb6ba3eea4d"
server_key_version: "encryptionkey@0"
I20260925 15:16:09.340798 26669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:16:09.357529 26669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:16:09.361958 26669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:16:09.362290 26669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:16:09.380906 26669 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.62:33201
I20260925 15:16:09.380926 26736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.62:33201 every 8 connection(s)
I20260925 15:16:09.382445 26669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
I20260925 15:16:09.386761 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 26669
I20260925 15:16:09.387741 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.1:44123
--local_ip_for_outbound_sockets=127.12.8.1
--tserver_master_addrs=127.12.8.62:33201
--webserver_port=38723
--webserver_interface=127.12.8.1
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.1.keytab
--principal=kudu/127.12.8.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:35145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:16:09.390641 26737 sys_catalog.cc:263] Verifying existing consensus state
I20260925 15:16:09.395973 26737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 594bdcb833214f8ca0ecc6906c3c8e64: Bootstrap starting.
I20260925 15:16:09.415576 26737 log.cc:824] T 00000000000000000000000000000000 P 594bdcb833214f8ca0ecc6906c3c8e64: Log is configured to *not* fsync() on all Append() calls
I20260925 15:16:09.427765 26737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 594bdcb833214f8ca0ecc6906c3c8e64: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 15:16:09.428296 26737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 594bdcb833214f8ca0ecc6906c3c8e64: Bootstrap complete.
I20260925 15:16:09.437495 26737 raft_consensus.cc:348] T 00000000000000000000000000000000 P 594bdcb833214f8ca0ecc6906c3c8e64 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "594bdcb833214f8ca0ecc6906c3c8e64" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 33201 } }
I20260925 15:16:09.438382 26737 raft_consensus.cc:729] T 00000000000000000000000000000000 P 594bdcb833214f8ca0ecc6906c3c8e64 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 594bdcb833214f8ca0ecc6906c3c8e64, State: Initialized, Role: FOLLOWER
I20260925 15:16:09.438970 26737 consensus_queue.cc:260] T 00000000000000000000000000000000 P 594bdcb833214f8ca0ecc6906c3c8e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "594bdcb833214f8ca0ecc6906c3c8e64" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 33201 } }
I20260925 15:16:09.439208 26737 raft_consensus.cc:388] T 00000000000000000000000000000000 P 594bdcb833214f8ca0ecc6906c3c8e64 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:16:09.439320 26737 raft_consensus.cc:482] T 00000000000000000000000000000000 P 594bdcb833214f8ca0ecc6906c3c8e64 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:16:09.439503 26737 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 594bdcb833214f8ca0ecc6906c3c8e64 [term 1 FOLLOWER]: Advancing to term 2
I20260925 15:16:09.443193 26737 raft_consensus.cc:504] T 00000000000000000000000000000000 P 594bdcb833214f8ca0ecc6906c3c8e64 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "594bdcb833214f8ca0ecc6906c3c8e64" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 33201 } }
I20260925 15:16:09.443691 26737 leader_election.cc:302] T 00000000000000000000000000000000 P 594bdcb833214f8ca0ecc6906c3c8e64 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 594bdcb833214f8ca0ecc6906c3c8e64; no voters:
I20260925 15:16:09.444363 26737 leader_election.cc:288] T 00000000000000000000000000000000 P 594bdcb833214f8ca0ecc6906c3c8e64 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 15:16:09.444525 26741 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 594bdcb833214f8ca0ecc6906c3c8e64 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 15:16:09.445804 26741 raft_consensus.cc:686] T 00000000000000000000000000000000 P 594bdcb833214f8ca0ecc6906c3c8e64 [term 2 LEADER]: Becoming Leader. State: Replica: 594bdcb833214f8ca0ecc6906c3c8e64, State: Running, Role: LEADER
I20260925 15:16:09.446599 26737 sys_catalog.cc:564] T 00000000000000000000000000000000 P 594bdcb833214f8ca0ecc6906c3c8e64 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:16:09.446455 26741 consensus_queue.cc:237] T 00000000000000000000000000000000 P 594bdcb833214f8ca0ecc6906c3c8e64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "594bdcb833214f8ca0ecc6906c3c8e64" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 33201 } }
I20260925 15:16:09.451753 26742 sys_catalog.cc:455] T 00000000000000000000000000000000 P 594bdcb833214f8ca0ecc6906c3c8e64 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "594bdcb833214f8ca0ecc6906c3c8e64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "594bdcb833214f8ca0ecc6906c3c8e64" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 33201 } } }
I20260925 15:16:09.452106 26742 sys_catalog.cc:458] T 00000000000000000000000000000000 P 594bdcb833214f8ca0ecc6906c3c8e64 [sys.catalog]: This master's current role is: LEADER
I20260925 15:16:09.452644 26743 sys_catalog.cc:455] T 00000000000000000000000000000000 P 594bdcb833214f8ca0ecc6906c3c8e64 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 594bdcb833214f8ca0ecc6906c3c8e64. Latest consensus state: current_term: 2 leader_uuid: "594bdcb833214f8ca0ecc6906c3c8e64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "594bdcb833214f8ca0ecc6906c3c8e64" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 33201 } } }
I20260925 15:16:09.452917 26743 sys_catalog.cc:458] T 00000000000000000000000000000000 P 594bdcb833214f8ca0ecc6906c3c8e64 [sys.catalog]: This master's current role is: LEADER
I20260925 15:16:09.461570 26749 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:16:09.469782 26749 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:16:09.471210 26749 catalog_manager.cc:1259] Loaded cluster ID: d0808ddb0aa64a32a4f15a30c19c15c3
I20260925 15:16:09.471366 26749 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:16:09.474596 26749 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260925 15:16:09.492475 26749 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260925 15:16:09.502235 26749 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:16:09.503997 26749 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 594bdcb833214f8ca0ecc6906c3c8e64: Loaded TSK: 0
I20260925 15:16:09.505466 26749 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 15:16:09.731846 26739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:16:09.732151 26739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:16:09.732332 26739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:16:09.739580 26739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:16:09.739738 26739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:16:09.739910 26739 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.1
I20260925 15:16:09.749624 26739 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:35145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.1.keytab
--principal=kudu/127.12.8.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.1:44123
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.8.1
--webserver_port=38723
--tserver_master_addrs=127.12.8.62:33201
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.1
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:16:09.750964 26739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:16:09.752254 26739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:16:09.766825 26765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:16:09.766925 26764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:16:09.768676 26739 server_base.cc:1034] running on GCE node
W20260925 15:16:09.769413 26767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:16:09.770069 26739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:16:09.771163 26739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:16:09.772393 26739 hybrid_clock.cc:648] HybridClock initialized: now 1790349369772346 us; error 39 us; skew 500 ppm
I20260925 15:16:09.772722 26739 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:16:09 dist-test-slave-p468 krb5kdc[26118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349369, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:16:09.776190 26739 init.cc:377] Logged in from keytab as kudu/127.12.8.1@KRBTEST.COM (short username kudu)
I20260925 15:16:09.777947 26739 webserver.cc:460] Webserver started at http://127.12.8.1:38723/ using document root <none> and password file <none>
I20260925 15:16:09.778820 26739 fs_manager.cc:359] Metadata directory not provided
I20260925 15:16:09.778973 26739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:16:09.786370 26739 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 15:16:09.789767 26774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:09.791468 26739 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:16:09.791636 26739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-0/wal
uuid: "e5c51475c8dd40cfb94ec868907836ca"
format_stamp: "Formatted at 2026-09-25 15:16:07 on dist-test-slave-p468"
server_key: "02203ba1aa34ee214db4ff6de2e8974d"
server_key_iv: "697a5fac1ab3e5e763f6fb0ef37ec69e"
server_key_version: "encryptionkey@0"
I20260925 15:16:09.792997 26739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:16:09.808789 26739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:16:09.813648 26739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:16:09.813913 26739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:16:09.814991 26739 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:16:09.816607 26739 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:16:09.816727 26739 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:09.816825 26739 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:16:09.816895 26739 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:09.857035 26739 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.1:44123
I20260925 15:16:09.857065 26886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.1:44123 every 8 connection(s)
I20260925 15:16:09.858978 26739 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-0/data/info.pb
I20260925 15:16:09.864187 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 26739
I20260925 15:16:09.864940 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.2:46647
--local_ip_for_outbound_sockets=127.12.8.2
--tserver_master_addrs=127.12.8.62:33201
--webserver_port=34027
--webserver_interface=127.12.8.2
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.2.keytab
--principal=kudu/127.12.8.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:35145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 15:16:09 dist-test-slave-p468 krb5kdc[26118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349369, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.1@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
I20260925 15:16:09.898169 26890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:09.866456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:33201 (local address 127.12.8.1:59139)
0925 15:16:09.878731 (+ 12275us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:09.878772 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:09.880554 (+ 1782us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:09.881952 (+ 1398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:09.881965 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:09.882606 (+ 641us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:16:09.883136 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:09.883164 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:09.884781 (+ 1617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:09.884786 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:09.886256 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:09.886269 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:09.887300 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:09.887308 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:09.887577 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:09.888271 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:16:09.888293 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:16:09.890492 (+ 2199us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:16:09.893838 (+ 3346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:16:09.893846 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:16:09.893860 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:16:09.894349 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:16:09.894808 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:16:09.894813 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:16:09.894817 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:16:09.895001 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:16:09.895773 (+ 772us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:16:09.895781 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:16:09.896105 (+ 324us) client_negotiation.cc:769] Sending connection context
0925 15:16:09.896369 (+ 264us) client_negotiation.cc:240] Negotiation successful
0925 15:16:09.896879 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12006,"thread_start_us":11812,"threads_started":1}
I20260925 15:16:09.898151 26889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:09.863416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.1:59139 (local address 127.12.8.62:33201)
0925 15:16:09.875624 (+ 12208us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:09.875642 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:09.878924 (+ 3282us) server_negotiation.cc:407] Connection header received
0925 15:16:09.881093 (+ 2169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:09.881106 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:09.881433 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:09.881803 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:09.883380 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:09.884605 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:09.886405 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:09.887054 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:09.887272 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:09.890775 (+ 3503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:16:09.890782 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:16:09.890796 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:16:09.890833 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:16:09.893684 (+ 2851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:09.894493 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:09.894508 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:09.894513 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:09.894684 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:09.895142 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:09.895147 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:09.895151 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:09.895618 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:16:09.895827 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:09.896587 (+ 760us) server_negotiation.cc:299] Negotiation successful
0925 15:16:09.896978 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11921,"thread_start_us":11782,"threads_started":1}
I20260925 15:16:09.901453 26887 heartbeater.cc:347] Connected to a master server at 127.12.8.62:33201
I20260925 15:16:09.901765 26887 heartbeater.cc:464] Registering TS with master...
I20260925 15:16:09.902541 26887 heartbeater.cc:511] Master 127.12.8.62:33201 requested a full tablet report, sending...
I20260925 15:16:09.904749 26702 ts_manager.cc:194] Registered new tserver with Master: e5c51475c8dd40cfb94ec868907836ca (127.12.8.1:44123)
I20260925 15:16:09.906312 26702 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.1@KRBTEST.COM'} at 127.12.8.1:59139
W20260925 15:16:10.192309 26891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:16:10.192559 26891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:16:10.192714 26891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:16:10.199945 26891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:16:10.200047 26891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:16:10.200230 26891 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.2
I20260925 15:16:10.209750 26891 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:35145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.2.keytab
--principal=kudu/127.12.8.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.2:46647
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.8.2
--webserver_port=34027
--tserver_master_addrs=127.12.8.62:33201
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.2
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:16:10.211016 26891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:16:10.212265 26891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:16:10.225062 26896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:16:10.225152 26899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:16:10.225183 26897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:16:10.225874 26891 server_base.cc:1034] running on GCE node
I20260925 15:16:10.226517 26891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:16:10.227406 26891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:16:10.228631 26891 hybrid_clock.cc:648] HybridClock initialized: now 1790349370228597 us; error 48 us; skew 500 ppm
I20260925 15:16:10.228951 26891 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:16:10 dist-test-slave-p468 krb5kdc[26118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349370, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:16:10.231961 26891 init.cc:377] Logged in from keytab as kudu/127.12.8.2@KRBTEST.COM (short username kudu)
I20260925 15:16:10.233274 26891 webserver.cc:460] Webserver started at http://127.12.8.2:34027/ using document root <none> and password file <none>
I20260925 15:16:10.233922 26891 fs_manager.cc:359] Metadata directory not provided
I20260925 15:16:10.234074 26891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:16:10.239243 26891 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 15:16:10.241780 26906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:10.243108 26891 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 15:16:10.243276 26891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-1/wal
uuid: "1b034ca722914ac1ac4b23691a36c014"
format_stamp: "Formatted at 2026-09-25 15:16:07 on dist-test-slave-p468"
server_key: "18f60b2de7d1d18d7fc5dca16b0ab3ee"
server_key_iv: "282252730ca62333142d9e7f224eafdc"
server_key_version: "encryptionkey@0"
I20260925 15:16:10.244225 26891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:16:10.279072 26891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:16:10.282974 26891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:16:10.283242 26891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:16:10.284216 26891 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:16:10.285501 26891 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:16:10.285624 26891 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:10.285758 26891 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:16:10.285852 26891 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:10.318410 26891 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.2:46647
I20260925 15:16:10.318457 27018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.2:46647 every 8 connection(s)
I20260925 15:16:10.319836 26891 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-1/data/info.pb
I20260925 15:16:10.322504 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 26891
I20260925 15:16:10.327105 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.3:41841
--local_ip_for_outbound_sockets=127.12.8.3
--tserver_master_addrs=127.12.8.62:33201
--webserver_port=32787
--webserver_interface=127.12.8.3
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.3.keytab
--principal=kudu/127.12.8.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:35145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 15:16:10 dist-test-slave-p468 krb5kdc[26118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349370, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.2@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
I20260925 15:16:10.340708 26889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:10.322369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.2:49517 (local address 127.12.8.62:33201)
0925 15:16:10.322568 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:10.322574 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:10.323846 (+ 1272us) server_negotiation.cc:407] Connection header received
0925 15:16:10.325641 (+ 1795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:10.325646 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:10.325740 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:10.325852 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:10.327471 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:10.328653 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:10.330229 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:10.331000 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:10.331152 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:10.334714 (+ 3562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:16:10.334719 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:16:10.334722 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:16:10.334760 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:16:10.337321 (+ 2561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:10.338300 (+ 979us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:10.338304 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:10.338307 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:10.338429 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:10.338937 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:10.338940 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:10.338942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:10.339201 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:16:10.339317 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:10.340264 (+ 947us) server_negotiation.cc:299] Negotiation successful
0925 15:16:10.340451 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 15:16:10.342165 27021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:10.322882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:33201 (local address 127.12.8.2:49517)
0925 15:16:10.323687 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:10.323731 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:10.325381 (+ 1650us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:10.326086 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:10.326094 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:10.326624 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:16:10.327272 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:10.327297 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:10.328829 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:10.328833 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:10.330044 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:10.330056 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:10.331205 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:10.331210 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:10.331400 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:10.332057 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:16:10.332082 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:16:10.334503 (+ 2421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:16:10.337467 (+ 2964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:16:10.337478 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:16:10.337497 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:16:10.338114 (+ 617us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:16:10.338537 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:16:10.338545 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:16:10.338552 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:16:10.338795 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:16:10.339333 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:16:10.339344 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:16:10.339864 (+ 520us) client_negotiation.cc:769] Sending connection context
0925 15:16:10.340163 (+ 299us) client_negotiation.cc:240] Negotiation successful
0925 15:16:10.340759 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":347,"threads_started":1}
I20260925 15:16:10.343793 27019 heartbeater.cc:347] Connected to a master server at 127.12.8.62:33201
I20260925 15:16:10.344097 27019 heartbeater.cc:464] Registering TS with master...
I20260925 15:16:10.344992 27019 heartbeater.cc:511] Master 127.12.8.62:33201 requested a full tablet report, sending...
I20260925 15:16:10.346586 26702 ts_manager.cc:194] Registered new tserver with Master: 1b034ca722914ac1ac4b23691a36c014 (127.12.8.2:46647)
I20260925 15:16:10.347570 26702 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.2@KRBTEST.COM'} at 127.12.8.2:49517
W20260925 15:16:10.654090 27022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:16:10.654399 27022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:16:10.654587 27022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:16:10.661840 27022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:16:10.661960 27022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:16:10.662141 27022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.3
I20260925 15:16:10.671684 27022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:35145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.3.keytab
--principal=kudu/127.12.8.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.3:41841
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.8.3
--webserver_port=32787
--tserver_master_addrs=127.12.8.62:33201
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.3
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:16:10.672905 27022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:16:10.674214 27022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:16:10.686970 27028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:16:10.687028 27027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:16:10.687048 27030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:16:10.688161 27022 server_base.cc:1034] running on GCE node
I20260925 15:16:10.688777 27022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:16:10.689711 27022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:16:10.690908 27022 hybrid_clock.cc:648] HybridClock initialized: now 1790349370690842 us; error 62 us; skew 500 ppm
I20260925 15:16:10.691217 27022 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:16:10 dist-test-slave-p468 krb5kdc[26118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349370, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:16:10.694247 27022 init.cc:377] Logged in from keytab as kudu/127.12.8.3@KRBTEST.COM (short username kudu)
I20260925 15:16:10.695667 27022 webserver.cc:460] Webserver started at http://127.12.8.3:32787/ using document root <none> and password file <none>
I20260925 15:16:10.696316 27022 fs_manager.cc:359] Metadata directory not provided
I20260925 15:16:10.696468 27022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:16:10.701442 27022 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 15:16:10.704005 27037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:10.705399 27022 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 15:16:10.705569 27022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-2/wal
uuid: "33c2d12159954b88ae9974464003472f"
format_stamp: "Formatted at 2026-09-25 15:16:07 on dist-test-slave-p468"
server_key: "eb96f366841014226ddfe887ce681d42"
server_key_iv: "24329e77fbeb1b69e7aaa732ed670f0c"
server_key_version: "encryptionkey@0"
I20260925 15:16:10.706574 27022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:16:10.728392 27022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:16:10.732538 27022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:16:10.732820 27022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:16:10.733747 27022 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:16:10.735116 27022 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:16:10.735266 27022 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:10.735410 27022 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:16:10.735499 27022 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:10.768956 27022 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.3:41841
I20260925 15:16:10.769040 27149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.3:41841 every 8 connection(s)
I20260925 15:16:10.770905 27022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-2/data/info.pb
I20260925 15:16:10.771590 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 27022
I20260925 15:16:10.798679 26889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:10.772945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53370 (local address 127.12.8.62:33201)
0925 15:16:10.773141 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:10.773148 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:10.784991 (+ 11843us) server_negotiation.cc:407] Connection header received
0925 15:16:10.785271 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:10.785280 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:10.785396 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:10.785539 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:10.787760 (+ 2221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:10.788477 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:10.789757 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:10.790418 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:10.790538 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:10.793259 (+ 2721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:16:10.793263 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:16:10.793267 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:16:10.793298 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:16:10.795977 (+ 2679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:10.796626 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:10.796638 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:10.796642 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:10.796771 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:10.797145 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:10.797151 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:10.797154 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:10.797485 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:16:10.797761 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:10.798071 (+ 310us) server_negotiation.cc:299] Negotiation successful
0925 15:16:10.798321 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
Sep 25 15:16:10 dist-test-slave-p468 krb5kdc[26118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349370, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.3@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
I20260925 15:16:10.806643 27153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:10.774386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.3:38983 (local address 127.12.8.62:33201)
0925 15:16:10.786534 (+ 12148us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:10.786543 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:10.790492 (+ 3949us) server_negotiation.cc:407] Connection header received
0925 15:16:10.792582 (+ 2090us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:10.792590 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:10.792678 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:10.792791 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:10.794579 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:10.795479 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:10.797008 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:10.797534 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:10.797680 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:10.801151 (+ 3471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:16:10.801156 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:16:10.801159 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:16:10.801187 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:16:10.803739 (+ 2552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:10.804427 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:10.804432 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:10.804435 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:10.804546 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:10.804985 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:10.804990 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:10.804993 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:10.805241 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:16:10.805400 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:10.806025 (+ 625us) server_negotiation.cc:299] Negotiation successful
0925 15:16:10.806260 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11993,"thread_start_us":12167,"threads_started":1}
I20260925 15:16:10.807495 27154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:10.774331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:33201 (local address 127.12.8.3:38983)
0925 15:16:10.790299 (+ 15968us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:10.790362 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:10.792265 (+ 1903us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:10.793075 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:10.793086 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:10.793632 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:16:10.794393 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:10.794413 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:10.795651 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:10.795659 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:10.796854 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:10.796866 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:10.797804 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:10.797813 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:10.798011 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:10.798852 (+ 841us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:16:10.798889 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:16:10.800941 (+ 2052us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:16:10.803885 (+ 2944us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:16:10.803894 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:16:10.803908 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:16:10.804304 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:16:10.804653 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:16:10.804658 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:16:10.804662 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:16:10.804868 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:16:10.805375 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:16:10.805383 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:16:10.805683 (+ 300us) client_negotiation.cc:769] Sending connection context
0925 15:16:10.805937 (+ 254us) client_negotiation.cc:240] Negotiation successful
0925 15:16:10.806446 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15624,"thread_start_us":18123,"threads_started":1}
I20260925 15:16:10.808696 27150 heartbeater.cc:347] Connected to a master server at 127.12.8.62:33201
I20260925 15:16:10.808952 27150 heartbeater.cc:464] Registering TS with master...
I20260925 15:16:10.809652 27150 heartbeater.cc:511] Master 127.12.8.62:33201 requested a full tablet report, sending...
I20260925 15:16:10.811120 26702 ts_manager.cc:194] Registered new tserver with Master: 33c2d12159954b88ae9974464003472f (127.12.8.3:41841)
I20260925 15:16:10.812047 26702 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.3@KRBTEST.COM'} at 127.12.8.3:38983
I20260925 15:16:10.813277 12320 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:16:10.819727 27153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:10.814137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53374 (local address 127.12.8.62:33201)
0925 15:16:10.814368 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:10.814376 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:10.814810 (+ 434us) server_negotiation.cc:407] Connection header received
0925 15:16:10.814989 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:10.814993 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:10.815135 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:10.815247 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:16:10.815655 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:10.816314 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:10.817781 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:10.818150 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:10.818309 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:10.818727 (+ 418us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:16:10.819167 (+ 440us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:16:10.819268 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:10.819388 (+ 120us) server_negotiation.cc:299] Negotiation successful
0925 15:16:10.819447 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260925 15:16:10.820627 26702 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:53374:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:16:10.827656 26702 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:16:10.851346 27161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:10.839139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49392 (local address 127.12.8.3:41841)
0925 15:16:10.841289 (+ 2150us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:10.841299 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:10.841332 (+ 33us) server_negotiation.cc:407] Connection header received
0925 15:16:10.841442 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:10.841449 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:10.841689 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:10.842712 (+ 1023us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:16:10.843222 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:10.844865 (+ 1643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:10.848001 (+ 3136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:10.849617 (+ 1616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:10.849834 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:10.850093 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:10.850803 (+ 710us) server_negotiation.cc:299] Negotiation successful
0925 15:16:10.850970 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1986,"thread_start_us":1838,"threads_started":1}
I20260925 15:16:10.851410 27162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:10.838969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.3:41841 (local address 127.0.0.1:49392)
0925 15:16:10.840133 (+ 1164us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:10.840168 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:10.840327 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:10.841932 (+ 1605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:10.841939 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:10.841969 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:16:10.842485 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:10.842501 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:10.845014 (+ 2513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:10.845021 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:10.847834 (+ 2813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:10.847849 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:10.850346 (+ 2497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:10.850353 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:10.850569 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:10.850613 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 15:16:10.850753 (+ 140us) client_negotiation.cc:240] Negotiation successful
0925 15:16:10.851006 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":905,"mutex_wait_us":112,"thread_start_us":2071,"threads_started":1}
I20260925 15:16:10.851408 27163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:10.839697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54298 (local address 127.12.8.2:46647)
0925 15:16:10.840871 (+ 1174us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:10.840881 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:10.841579 (+ 698us) server_negotiation.cc:407] Connection header received
0925 15:16:10.841678 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:10.841684 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:10.841887 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:10.842070 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:16:10.843026 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:10.844808 (+ 1782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:10.848403 (+ 3595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:10.849873 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:10.849980 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:10.850144 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:10.850923 (+ 779us) server_negotiation.cc:299] Negotiation successful
0925 15:16:10.851094 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1023,"thread_start_us":910,"threads_started":1}
I20260925 15:16:10.852161 27160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:10.839086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.2:46647 (local address 127.0.0.1:54298)
0925 15:16:10.841320 (+ 2234us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:10.841336 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:10.841453 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:10.842378 (+ 925us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:10.842385 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:10.842414 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:16:10.842801 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:10.842814 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:10.844988 (+ 2174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:10.844994 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:10.848222 (+ 3228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:10.848233 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:10.850417 (+ 2184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:10.850424 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:10.850657 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:10.850705 (+ 48us) client_negotiation.cc:769] Sending connection context
0925 15:16:10.850807 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 15:16:10.851768 (+ 961us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2112,"mutex_wait_us":106,"thread_start_us":795,"threads_started":1}
I20260925 15:16:10.855527 27164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:10.841516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.1:44123 (local address 127.0.0.1:60190)
0925 15:16:10.845910 (+ 4394us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:10.845928 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:10.846028 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:10.848212 (+ 2184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:10.848217 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:10.848243 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:16:10.848639 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:10.848649 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:10.850456 (+ 1807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:10.850460 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:10.852577 (+ 2117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:10.852591 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:10.854733 (+ 2142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:10.854739 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:10.854928 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:10.854962 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 15:16:10.855059 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 15:16:10.855225 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4253,"thread_start_us":4110,"threads_started":1}
I20260925 15:16:10.855656 27165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:10.841875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60190 (local address 127.12.8.1:44123)
0925 15:16:10.847506 (+ 5631us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:10.847512 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:10.847535 (+ 23us) server_negotiation.cc:407] Connection header received
0925 15:16:10.847648 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:10.847658 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:10.848011 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:10.848232 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:16:10.848821 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:10.850335 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:10.852738 (+ 2403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:10.854161 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:10.854312 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:10.854503 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:10.855137 (+ 634us) server_negotiation.cc:299] Negotiation successful
0925 15:16:10.855311 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5480,"thread_start_us":5322,"threads_started":1}
I20260925 15:16:10.855578 27085 tablet_service.cc:1467] Processing CreateTablet for tablet 795c8f91e19941e0bfe74be9667758c2 (DEFAULT_TABLE table=test-table [id=654089449e0941fb8eabdb84d8595ff8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:16:10.855576 26954 tablet_service.cc:1467] Processing CreateTablet for tablet 795c8f91e19941e0bfe74be9667758c2 (DEFAULT_TABLE table=test-table [id=654089449e0941fb8eabdb84d8595ff8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:16:10.857072 27085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 795c8f91e19941e0bfe74be9667758c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:16:10.857080 26954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 795c8f91e19941e0bfe74be9667758c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:16:10.859630 26822 tablet_service.cc:1467] Processing CreateTablet for tablet 795c8f91e19941e0bfe74be9667758c2 (DEFAULT_TABLE table=test-table [id=654089449e0941fb8eabdb84d8595ff8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:16:10.860994 26822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 795c8f91e19941e0bfe74be9667758c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:16:10.871474 27166 tablet_bootstrap.cc:492] T 795c8f91e19941e0bfe74be9667758c2 P 33c2d12159954b88ae9974464003472f: Bootstrap starting.
I20260925 15:16:10.874787 27166 tablet_bootstrap.cc:654] T 795c8f91e19941e0bfe74be9667758c2 P 33c2d12159954b88ae9974464003472f: Neither blocks nor log segments found. Creating new log.
I20260925 15:16:10.875762 27166 log.cc:824] T 795c8f91e19941e0bfe74be9667758c2 P 33c2d12159954b88ae9974464003472f: Log is configured to *not* fsync() on all Append() calls
I20260925 15:16:10.878330 27167 tablet_bootstrap.cc:492] T 795c8f91e19941e0bfe74be9667758c2 P 1b034ca722914ac1ac4b23691a36c014: Bootstrap starting.
I20260925 15:16:10.878551 27166 tablet_bootstrap.cc:492] T 795c8f91e19941e0bfe74be9667758c2 P 33c2d12159954b88ae9974464003472f: No bootstrap required, opened a new log
I20260925 15:16:10.878832 27166 ts_tablet_manager.cc:1397] T 795c8f91e19941e0bfe74be9667758c2 P 33c2d12159954b88ae9974464003472f: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260925 15:16:10.879329 27169 tablet_bootstrap.cc:492] T 795c8f91e19941e0bfe74be9667758c2 P e5c51475c8dd40cfb94ec868907836ca: Bootstrap starting.
I20260925 15:16:10.881742 27167 tablet_bootstrap.cc:654] T 795c8f91e19941e0bfe74be9667758c2 P 1b034ca722914ac1ac4b23691a36c014: Neither blocks nor log segments found. Creating new log.
I20260925 15:16:10.882655 27169 tablet_bootstrap.cc:654] T 795c8f91e19941e0bfe74be9667758c2 P e5c51475c8dd40cfb94ec868907836ca: Neither blocks nor log segments found. Creating new log.
I20260925 15:16:10.883023 27167 log.cc:824] T 795c8f91e19941e0bfe74be9667758c2 P 1b034ca722914ac1ac4b23691a36c014: Log is configured to *not* fsync() on all Append() calls
I20260925 15:16:10.883809 27169 log.cc:824] T 795c8f91e19941e0bfe74be9667758c2 P e5c51475c8dd40cfb94ec868907836ca: Log is configured to *not* fsync() on all Append() calls
I20260925 15:16:10.885972 27167 tablet_bootstrap.cc:492] T 795c8f91e19941e0bfe74be9667758c2 P 1b034ca722914ac1ac4b23691a36c014: No bootstrap required, opened a new log
I20260925 15:16:10.886270 27167 ts_tablet_manager.cc:1397] T 795c8f91e19941e0bfe74be9667758c2 P 1b034ca722914ac1ac4b23691a36c014: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.005s
I20260925 15:16:10.888285 27169 tablet_bootstrap.cc:492] T 795c8f91e19941e0bfe74be9667758c2 P e5c51475c8dd40cfb94ec868907836ca: No bootstrap required, opened a new log
I20260925 15:16:10.888532 27169 ts_tablet_manager.cc:1397] T 795c8f91e19941e0bfe74be9667758c2 P e5c51475c8dd40cfb94ec868907836ca: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20260925 15:16:10.889178 27166 raft_consensus.cc:348] T 795c8f91e19941e0bfe74be9667758c2 P 33c2d12159954b88ae9974464003472f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33c2d12159954b88ae9974464003472f" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 41841 } } peers { permanent_uuid: "1b034ca722914ac1ac4b23691a36c014" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 46647 } } peers { permanent_uuid: "e5c51475c8dd40cfb94ec868907836ca" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 44123 } }
I20260925 15:16:10.889592 27166 raft_consensus.cc:374] T 795c8f91e19941e0bfe74be9667758c2 P 33c2d12159954b88ae9974464003472f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:16:10.889729 27166 raft_consensus.cc:729] T 795c8f91e19941e0bfe74be9667758c2 P 33c2d12159954b88ae9974464003472f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33c2d12159954b88ae9974464003472f, State: Initialized, Role: FOLLOWER
I20260925 15:16:10.890424 27166 consensus_queue.cc:260] T 795c8f91e19941e0bfe74be9667758c2 P 33c2d12159954b88ae9974464003472f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33c2d12159954b88ae9974464003472f" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 41841 } } peers { permanent_uuid: "1b034ca722914ac1ac4b23691a36c014" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 46647 } } peers { permanent_uuid: "e5c51475c8dd40cfb94ec868907836ca" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 44123 } }
I20260925 15:16:10.894912 27150 heartbeater.cc:503] Master 127.12.8.62:33201 was elected leader, sending a full tablet report...
I20260925 15:16:10.895591 27166 ts_tablet_manager.cc:1428] T 795c8f91e19941e0bfe74be9667758c2 P 33c2d12159954b88ae9974464003472f: Time spent starting tablet: real 0.017s user 0.012s sys 0.004s
I20260925 15:16:10.896514 27169 raft_consensus.cc:348] T 795c8f91e19941e0bfe74be9667758c2 P e5c51475c8dd40cfb94ec868907836ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33c2d12159954b88ae9974464003472f" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 41841 } } peers { permanent_uuid: "1b034ca722914ac1ac4b23691a36c014" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 46647 } } peers { permanent_uuid: "e5c51475c8dd40cfb94ec868907836ca" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 44123 } }
I20260925 15:16:10.896883 27169 raft_consensus.cc:374] T 795c8f91e19941e0bfe74be9667758c2 P e5c51475c8dd40cfb94ec868907836ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:16:10.897004 27169 raft_consensus.cc:729] T 795c8f91e19941e0bfe74be9667758c2 P e5c51475c8dd40cfb94ec868907836ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5c51475c8dd40cfb94ec868907836ca, State: Initialized, Role: FOLLOWER
I20260925 15:16:10.897644 27169 consensus_queue.cc:260] T 795c8f91e19941e0bfe74be9667758c2 P e5c51475c8dd40cfb94ec868907836ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33c2d12159954b88ae9974464003472f" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 41841 } } peers { permanent_uuid: "1b034ca722914ac1ac4b23691a36c014" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 46647 } } peers { permanent_uuid: "e5c51475c8dd40cfb94ec868907836ca" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 44123 } }
I20260925 15:16:10.897612 27167 raft_consensus.cc:348] T 795c8f91e19941e0bfe74be9667758c2 P 1b034ca722914ac1ac4b23691a36c014 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33c2d12159954b88ae9974464003472f" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 41841 } } peers { permanent_uuid: "1b034ca722914ac1ac4b23691a36c014" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 46647 } } peers { permanent_uuid: "e5c51475c8dd40cfb94ec868907836ca" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 44123 } }
I20260925 15:16:10.898078 27167 raft_consensus.cc:374] T 795c8f91e19941e0bfe74be9667758c2 P 1b034ca722914ac1ac4b23691a36c014 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:16:10.898250 27167 raft_consensus.cc:729] T 795c8f91e19941e0bfe74be9667758c2 P 1b034ca722914ac1ac4b23691a36c014 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b034ca722914ac1ac4b23691a36c014, State: Initialized, Role: FOLLOWER
I20260925 15:16:10.898823 26887 heartbeater.cc:503] Master 127.12.8.62:33201 was elected leader, sending a full tablet report...
I20260925 15:16:10.898902 27167 consensus_queue.cc:260] T 795c8f91e19941e0bfe74be9667758c2 P 1b034ca722914ac1ac4b23691a36c014 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33c2d12159954b88ae9974464003472f" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 41841 } } peers { permanent_uuid: "1b034ca722914ac1ac4b23691a36c014" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 46647 } } peers { permanent_uuid: "e5c51475c8dd40cfb94ec868907836ca" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 44123 } }
I20260925 15:16:10.899509 27169 ts_tablet_manager.cc:1428] T 795c8f91e19941e0bfe74be9667758c2 P e5c51475c8dd40cfb94ec868907836ca: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260925 15:16:10.900384 27019 heartbeater.cc:503] Master 127.12.8.62:33201 was elected leader, sending a full tablet report...
I20260925 15:16:10.900954 27167 ts_tablet_manager.cc:1428] T 795c8f91e19941e0bfe74be9667758c2 P 1b034ca722914ac1ac4b23691a36c014: Time spent starting tablet: real 0.014s user 0.010s sys 0.003s
I20260925 15:16:11.018937 27172 raft_consensus.cc:482] T 795c8f91e19941e0bfe74be9667758c2 P 33c2d12159954b88ae9974464003472f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:16:11.019277 27172 raft_consensus.cc:504] T 795c8f91e19941e0bfe74be9667758c2 P 33c2d12159954b88ae9974464003472f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33c2d12159954b88ae9974464003472f" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 41841 } } peers { permanent_uuid: "1b034ca722914ac1ac4b23691a36c014" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 46647 } } peers { permanent_uuid: "e5c51475c8dd40cfb94ec868907836ca" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 44123 } }
I20260925 15:16:11.021267 27172 leader_election.cc:288] T 795c8f91e19941e0bfe74be9667758c2 P 33c2d12159954b88ae9974464003472f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b034ca722914ac1ac4b23691a36c014 (127.12.8.2:46647), e5c51475c8dd40cfb94ec868907836ca (127.12.8.1:44123)
I20260925 15:16:11.028266 27163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:11.021715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.3:48279 (local address 127.12.8.2:46647)
0925 15:16:11.022155 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:11.022163 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:11.022210 (+ 47us) server_negotiation.cc:407] Connection header received
0925 15:16:11.022383 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:11.022390 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:11.022480 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:11.022587 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:16:11.023075 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:11.024090 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:11.026276 (+ 2186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:11.027244 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:11.027347 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:11.027415 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:11.027867 (+ 452us) server_negotiation.cc:299] Negotiation successful
0925 15:16:11.027964 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293}
I20260925 15:16:11.028293 27154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:11.021688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.2:46647 (local address 127.12.8.3:48279)
0925 15:16:11.021963 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:11.021988 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:11.022139 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:11.022618 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:11.022622 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:11.022647 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:16:11.022881 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:11.022893 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:11.024268 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:11.024275 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:11.026065 (+ 1790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:11.026087 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:11.027416 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:11.027423 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:11.027641 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:11.027681 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 15:16:11.027805 (+ 124us) client_negotiation.cc:240] Negotiation successful
0925 15:16:11.027943 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106}
I20260925 15:16:11.028973 27165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:11.021906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.3:48561 (local address 127.12.8.1:44123)
0925 15:16:11.022102 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:11.022109 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:11.022771 (+ 662us) server_negotiation.cc:407] Connection header received
0925 15:16:11.022943 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:11.022950 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:11.023039 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:11.023114 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:16:11.023792 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:11.024902 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:11.026813 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:11.027942 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:11.028086 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:11.028198 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:11.028565 (+ 367us) server_negotiation.cc:299] Negotiation successful
0925 15:16:11.028700 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 15:16:11.029371 27175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:11.021805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.1:44123 (local address 127.12.8.3:48561)
0925 15:16:11.022658 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:11.022675 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:11.022797 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:11.023176 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:11.023181 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:11.023207 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:16:11.023576 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:11.023587 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:11.025059 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:11.025063 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:11.026654 (+ 1591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:11.026670 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:11.028167 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:11.028174 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:11.028361 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:11.028399 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 15:16:11.028882 (+ 483us) client_negotiation.cc:240] Negotiation successful
0925 15:16:11.029015 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":712,"mutex_wait_us":3,"thread_start_us":545,"threads_started":1}
I20260925 15:16:11.029450 26974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "795c8f91e19941e0bfe74be9667758c2" candidate_uuid: "33c2d12159954b88ae9974464003472f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b034ca722914ac1ac4b23691a36c014" is_pre_election: true
I20260925 15:16:11.030023 26974 raft_consensus.cc:2454] T 795c8f91e19941e0bfe74be9667758c2 P 1b034ca722914ac1ac4b23691a36c014 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33c2d12159954b88ae9974464003472f in term 0.
I20260925 15:16:11.030426 26842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "795c8f91e19941e0bfe74be9667758c2" candidate_uuid: "33c2d12159954b88ae9974464003472f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5c51475c8dd40cfb94ec868907836ca" is_pre_election: true
I20260925 15:16:11.030854 27039 leader_election.cc:302] T 795c8f91e19941e0bfe74be9667758c2 P 33c2d12159954b88ae9974464003472f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b034ca722914ac1ac4b23691a36c014, 33c2d12159954b88ae9974464003472f; no voters:
I20260925 15:16:11.031004 26842 raft_consensus.cc:2454] T 795c8f91e19941e0bfe74be9667758c2 P e5c51475c8dd40cfb94ec868907836ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33c2d12159954b88ae9974464003472f in term 0.
I20260925 15:16:11.031280 27172 raft_consensus.cc:2781] T 795c8f91e19941e0bfe74be9667758c2 P 33c2d12159954b88ae9974464003472f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:16:11.031435 27172 raft_consensus.cc:482] T 795c8f91e19941e0bfe74be9667758c2 P 33c2d12159954b88ae9974464003472f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:16:11.031558 27172 raft_consensus.cc:3037] T 795c8f91e19941e0bfe74be9667758c2 P 33c2d12159954b88ae9974464003472f [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:16:11.033775 27172 raft_consensus.cc:504] T 795c8f91e19941e0bfe74be9667758c2 P 33c2d12159954b88ae9974464003472f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33c2d12159954b88ae9974464003472f" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 41841 } } peers { permanent_uuid: "1b034ca722914ac1ac4b23691a36c014" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 46647 } } peers { permanent_uuid: "e5c51475c8dd40cfb94ec868907836ca" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 44123 } }
I20260925 15:16:11.034407 27172 leader_election.cc:288] T 795c8f91e19941e0bfe74be9667758c2 P 33c2d12159954b88ae9974464003472f [CANDIDATE]: Term 1 election: Requested vote from peers 1b034ca722914ac1ac4b23691a36c014 (127.12.8.2:46647), e5c51475c8dd40cfb94ec868907836ca (127.12.8.1:44123)
I20260925 15:16:11.034735 26974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "795c8f91e19941e0bfe74be9667758c2" candidate_uuid: "33c2d12159954b88ae9974464003472f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b034ca722914ac1ac4b23691a36c014"
I20260925 15:16:11.034849 26842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "795c8f91e19941e0bfe74be9667758c2" candidate_uuid: "33c2d12159954b88ae9974464003472f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5c51475c8dd40cfb94ec868907836ca"
I20260925 15:16:11.034940 26974 raft_consensus.cc:3037] T 795c8f91e19941e0bfe74be9667758c2 P 1b034ca722914ac1ac4b23691a36c014 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:16:11.035043 26842 raft_consensus.cc:3037] T 795c8f91e19941e0bfe74be9667758c2 P e5c51475c8dd40cfb94ec868907836ca [term 0 FOLLOWER]: Advancing to term 1
W20260925 15:16:11.037667 27151 tablet.cc:2369] T 795c8f91e19941e0bfe74be9667758c2 P 33c2d12159954b88ae9974464003472f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:16:11.038230 26842 raft_consensus.cc:2454] T 795c8f91e19941e0bfe74be9667758c2 P e5c51475c8dd40cfb94ec868907836ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33c2d12159954b88ae9974464003472f in term 1.
I20260925 15:16:11.038230 26974 raft_consensus.cc:2454] T 795c8f91e19941e0bfe74be9667758c2 P 1b034ca722914ac1ac4b23691a36c014 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33c2d12159954b88ae9974464003472f in term 1.
I20260925 15:16:11.038604 27038 leader_election.cc:302] T 795c8f91e19941e0bfe74be9667758c2 P 33c2d12159954b88ae9974464003472f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33c2d12159954b88ae9974464003472f, e5c51475c8dd40cfb94ec868907836ca; no voters:
I20260925 15:16:11.038941 27172 raft_consensus.cc:2781] T 795c8f91e19941e0bfe74be9667758c2 P 33c2d12159954b88ae9974464003472f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:16:11.039397 27172 raft_consensus.cc:686] T 795c8f91e19941e0bfe74be9667758c2 P 33c2d12159954b88ae9974464003472f [term 1 LEADER]: Becoming Leader. State: Replica: 33c2d12159954b88ae9974464003472f, State: Running, Role: LEADER
I20260925 15:16:11.039909 27172 consensus_queue.cc:237] T 795c8f91e19941e0bfe74be9667758c2 P 33c2d12159954b88ae9974464003472f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33c2d12159954b88ae9974464003472f" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 41841 } } peers { permanent_uuid: "1b034ca722914ac1ac4b23691a36c014" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 46647 } } peers { permanent_uuid: "e5c51475c8dd40cfb94ec868907836ca" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 44123 } }
I20260925 15:16:11.044323 26702 catalog_manager.cc:5495] T 795c8f91e19941e0bfe74be9667758c2 P 33c2d12159954b88ae9974464003472f reported cstate change: term changed from 0 to 1, leader changed from <none> to 33c2d12159954b88ae9974464003472f (127.12.8.3). New cstate: current_term: 1 leader_uuid: "33c2d12159954b88ae9974464003472f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33c2d12159954b88ae9974464003472f" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 41841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b034ca722914ac1ac4b23691a36c014" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 46647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5c51475c8dd40cfb94ec868907836ca" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 44123 } health_report { overall_health: UNKNOWN } } }
W20260925 15:16:11.071749 27020 tablet.cc:2369] T 795c8f91e19941e0bfe74be9667758c2 P 1b034ca722914ac1ac4b23691a36c014: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:16:11.099907 27161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:11.094369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49402 (local address 127.12.8.3:41841)
0925 15:16:11.094600 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:11.094605 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:11.094623 (+ 18us) server_negotiation.cc:407] Connection header received
0925 15:16:11.094831 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:11.094835 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:11.094916 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:11.095030 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:16:11.095375 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:11.096071 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:11.097673 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:11.098047 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:11.098151 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:11.098825 (+ 674us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:16:11.099304 (+ 479us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:16:11.099461 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:11.099561 (+ 100us) server_negotiation.cc:299] Negotiation successful
0925 15:16:11.099636 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260925 15:16:11.107740 26974 raft_consensus.cc:1261] T 795c8f91e19941e0bfe74be9667758c2 P 1b034ca722914ac1ac4b23691a36c014 [term 1 FOLLOWER]: Refusing update from remote peer 33c2d12159954b88ae9974464003472f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:16:11.107740 26842 raft_consensus.cc:1261] T 795c8f91e19941e0bfe74be9667758c2 P e5c51475c8dd40cfb94ec868907836ca [term 1 FOLLOWER]: Refusing update from remote peer 33c2d12159954b88ae9974464003472f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:16:11.108667 27176 consensus_queue.cc:1035] T 795c8f91e19941e0bfe74be9667758c2 P 33c2d12159954b88ae9974464003472f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5c51475c8dd40cfb94ec868907836ca" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 44123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:16:11.108989 27172 consensus_queue.cc:1035] T 795c8f91e19941e0bfe74be9667758c2 P 33c2d12159954b88ae9974464003472f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b034ca722914ac1ac4b23691a36c014" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 46647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 15:16:11.119804 26888 tablet.cc:2369] T 795c8f91e19941e0bfe74be9667758c2 P e5c51475c8dd40cfb94ec868907836ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:16:11.119638 27179 mvcc.cc:205] Tried to move back new op lower bound from 7333271024047452160 to 7333271023782936576. Current Snapshot: MvccSnapshot[applied={T|T < 7333271024047452160}]
I20260925 15:16:11.132630 26702 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:53374:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:16:11.132863 26702 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:53374:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:16:11.136097 26702 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 594bdcb833214f8ca0ecc6906c3c8e64: Sending DeleteTablet for 3 replicas of tablet 795c8f91e19941e0bfe74be9667758c2
I20260925 15:16:11.137099 27085 tablet_service.cc:1514] Processing DeleteTablet for tablet 795c8f91e19941e0bfe74be9667758c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:16:11 UTC) from {username='kudu', principal='kudu/127.12.8.62@KRBTEST.COM'} at 127.0.0.1:49392
I20260925 15:16:11.137276 26954 tablet_service.cc:1514] Processing DeleteTablet for tablet 795c8f91e19941e0bfe74be9667758c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:16:11 UTC) from {username='kudu', principal='kudu/127.12.8.62@KRBTEST.COM'} at 127.0.0.1:54298
I20260925 15:16:11.138089 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 26739
I20260925 15:16:11.138031 26822 tablet_service.cc:1514] Processing DeleteTablet for tablet 795c8f91e19941e0bfe74be9667758c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:16:11 UTC) from {username='kudu', principal='kudu/127.12.8.62@KRBTEST.COM'} at 127.0.0.1:60190
I20260925 15:16:11.138265 27188 tablet_replica.cc:330] T 795c8f91e19941e0bfe74be9667758c2 P 33c2d12159954b88ae9974464003472f: stopping tablet replica
I20260925 15:16:11.138655 27188 raft_consensus.cc:2229] T 795c8f91e19941e0bfe74be9667758c2 P 33c2d12159954b88ae9974464003472f [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:16:11.139389 27188 raft_consensus.cc:2258] T 795c8f91e19941e0bfe74be9667758c2 P 33c2d12159954b88ae9974464003472f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:16:11.141518 27188 ts_tablet_manager.cc:1905] T 795c8f91e19941e0bfe74be9667758c2 P 33c2d12159954b88ae9974464003472f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:16:11.149032 27188 ts_tablet_manager.cc:1918] T 795c8f91e19941e0bfe74be9667758c2 P 33c2d12159954b88ae9974464003472f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:16:11.149240 27188 log.cc:1196] T 795c8f91e19941e0bfe74be9667758c2 P 33c2d12159954b88ae9974464003472f: Deleting WAL directory at /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-2/wal/wals/795c8f91e19941e0bfe74be9667758c2
I20260925 15:16:11.149825 27188 ts_tablet_manager.cc:1939] T 795c8f91e19941e0bfe74be9667758c2 P 33c2d12159954b88ae9974464003472f: Deleting consensus metadata
I20260925 15:16:11.150823 27189 tablet_replica.cc:330] T 795c8f91e19941e0bfe74be9667758c2 P 1b034ca722914ac1ac4b23691a36c014: stopping tablet replica
I20260925 15:16:11.151450 27189 raft_consensus.cc:2229] T 795c8f91e19941e0bfe74be9667758c2 P 1b034ca722914ac1ac4b23691a36c014 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:16:11.151541 26686 catalog_manager.cc:4841] TS 33c2d12159954b88ae9974464003472f (127.12.8.3:41841): tablet 795c8f91e19941e0bfe74be9667758c2 (table test-table [id=654089449e0941fb8eabdb84d8595ff8]) successfully deleted
I20260925 15:16:11.152108 27189 raft_consensus.cc:2258] T 795c8f91e19941e0bfe74be9667758c2 P 1b034ca722914ac1ac4b23691a36c014 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:16:11.152216 27190 tablet_replica.cc:330] T 795c8f91e19941e0bfe74be9667758c2 P e5c51475c8dd40cfb94ec868907836ca: stopping tablet replica
I20260925 15:16:11.152581 27190 raft_consensus.cc:2229] T 795c8f91e19941e0bfe74be9667758c2 P e5c51475c8dd40cfb94ec868907836ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:16:11.153082 27190 raft_consensus.cc:2258] T 795c8f91e19941e0bfe74be9667758c2 P e5c51475c8dd40cfb94ec868907836ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:16:11.154440 27189 ts_tablet_manager.cc:1905] T 795c8f91e19941e0bfe74be9667758c2 P 1b034ca722914ac1ac4b23691a36c014: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:16:11.155006 27190 ts_tablet_manager.cc:1905] T 795c8f91e19941e0bfe74be9667758c2 P e5c51475c8dd40cfb94ec868907836ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:16:11.161995 27189 ts_tablet_manager.cc:1918] T 795c8f91e19941e0bfe74be9667758c2 P 1b034ca722914ac1ac4b23691a36c014: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:16:11.162009 27190 ts_tablet_manager.cc:1918] T 795c8f91e19941e0bfe74be9667758c2 P e5c51475c8dd40cfb94ec868907836ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:16:11.162171 27190 log.cc:1196] T 795c8f91e19941e0bfe74be9667758c2 P e5c51475c8dd40cfb94ec868907836ca: Deleting WAL directory at /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-0/wal/wals/795c8f91e19941e0bfe74be9667758c2
I20260925 15:16:11.162210 27189 log.cc:1196] T 795c8f91e19941e0bfe74be9667758c2 P 1b034ca722914ac1ac4b23691a36c014: Deleting WAL directory at /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-1/wal/wals/795c8f91e19941e0bfe74be9667758c2
I20260925 15:16:11.162761 27189 ts_tablet_manager.cc:1939] T 795c8f91e19941e0bfe74be9667758c2 P 1b034ca722914ac1ac4b23691a36c014: Deleting consensus metadata
I20260925 15:16:11.162796 27190 ts_tablet_manager.cc:1939] T 795c8f91e19941e0bfe74be9667758c2 P e5c51475c8dd40cfb94ec868907836ca: Deleting consensus metadata
I20260925 15:16:11.163862 27165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:11.138967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60196 (local address 127.12.8.1:44123)
0925 15:16:11.139134 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:11.139140 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:11.139375 (+ 235us) server_negotiation.cc:407] Connection header received
0925 15:16:11.139570 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:11.139576 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:11.139671 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:11.139779 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:11.141842 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:11.143039 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:11.145586 (+ 2547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:11.146257 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:11.146373 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:11.155702 (+ 9329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:16:11.155715 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:16:11.155730 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:16:11.155774 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:16:11.159723 (+ 3949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:11.160615 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:11.160624 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:11.160628 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:11.160786 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:11.161219 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:11.161224 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:11.161228 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:11.161699 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:16:11.162533 (+ 834us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:11.163268 (+ 735us) server_negotiation.cc:299] Negotiation successful
0925 15:16:11.163507 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 15:16:11.164579 26687 catalog_manager.cc:4841] TS 1b034ca722914ac1ac4b23691a36c014 (127.12.8.2:46647): tablet 795c8f91e19941e0bfe74be9667758c2 (table test-table [id=654089449e0941fb8eabdb84d8595ff8]) successfully deleted
I20260925 15:16:11.164575 26686 catalog_manager.cc:4841] TS e5c51475c8dd40cfb94ec868907836ca (127.12.8.1:44123): tablet 795c8f91e19941e0bfe74be9667758c2 (table test-table [id=654089449e0941fb8eabdb84d8595ff8]) successfully deleted
I20260925 15:16:11.165499 26882 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:16:11.254660 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 26739
I20260925 15:16:11.274840 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 26891
I20260925 15:16:11.290896 27163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:11.275850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54302 (local address 127.12.8.2:46647)
0925 15:16:11.276067 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:11.276074 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:11.276097 (+ 23us) server_negotiation.cc:407] Connection header received
0925 15:16:11.276175 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:11.276181 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:11.276263 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:11.276369 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:11.277903 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:11.279092 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:11.280519 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:11.281154 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:11.281293 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:11.284565 (+ 3272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:16:11.284580 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:16:11.284594 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:16:11.284630 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:16:11.287557 (+ 2927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:11.288441 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:11.288448 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:11.288450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:11.288611 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:11.289133 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:11.289138 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:11.289140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:11.289578 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:16:11.289852 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:11.290281 (+ 429us) server_negotiation.cc:299] Negotiation successful
0925 15:16:11.290514 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 15:16:11.291695 27014 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:16:11.380820 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 26891
I20260925 15:16:11.405143 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 27022
I20260925 15:16:11.421372 27161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:11.406052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49408 (local address 127.12.8.3:41841)
0925 15:16:11.406284 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:11.406289 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:11.406316 (+ 27us) server_negotiation.cc:407] Connection header received
0925 15:16:11.406498 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:11.406504 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:11.406582 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:11.406672 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:11.408462 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:11.409334 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:11.411326 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:11.411767 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:11.411862 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:11.415256 (+ 3394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:16:11.415267 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:16:11.415289 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:16:11.415334 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:16:11.418516 (+ 3182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:11.419341 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:11.419347 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:11.419350 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:11.419504 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:11.419915 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:11.419918 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:11.419920 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:11.420241 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:16:11.420464 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:11.420827 (+ 363us) server_negotiation.cc:299] Negotiation successful
0925 15:16:11.420987 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 15:16:11.422032 27145 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:16:11.542572 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 27022
I20260925 15:16:11.561328 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 26669
I20260925 15:16:11.562578 26732 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:16:11.674046 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 26669
I20260925 15:16:11.695472 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.62:33201
--webserver_interface=127.12.8.62
--webserver_port=46131
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab
--principal=kudu/127.12.8.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:35145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.62:33201
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:16:12.015722 27195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:16:12.016016 27195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:16:12.016104 27195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:16:12.023262 27195 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:16:12.023363 27195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:16:12.023413 27195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:16:12.023470 27195 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:16:12.023514 27195 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:16:12.032900 27195 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:35145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.8.62:33201
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab
--principal=kudu/127.12.8.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.62:33201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.8.62
--webserver_port=46131
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:16:12.034104 27195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:16:12.035351 27195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:16:12.046403 27200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:16:12.046454 27201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:16:12.046432 27203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:16:12.047144 27195 server_base.cc:1034] running on GCE node
I20260925 15:16:12.047930 27195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:16:12.049302 27195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:16:12.050523 27195 hybrid_clock.cc:648] HybridClock initialized: now 1790349372050462 us; error 46 us; skew 500 ppm
I20260925 15:16:12.050864 27195 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:16:12 dist-test-slave-p468 krb5kdc[26118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349372, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:16:12.053877 27195 init.cc:377] Logged in from keytab as kudu/127.12.8.62@KRBTEST.COM (short username kudu)
I20260925 15:16:12.055392 27195 webserver.cc:460] Webserver started at http://127.12.8.62:46131/ using document root <none> and password file <none>
I20260925 15:16:12.056042 27195 fs_manager.cc:359] Metadata directory not provided
I20260925 15:16:12.056190 27195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:16:12.061269 27195 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260925 15:16:12.064113 27210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:12.065415 27195 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 15:16:12.065585 27195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/master-0/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/master-0/wal
uuid: "594bdcb833214f8ca0ecc6906c3c8e64"
format_stamp: "Formatted at 2026-09-25 15:16:06 on dist-test-slave-p468"
server_key: "21b8a2d0a9e431406afae1549ce7fdac"
server_key_iv: "2d27c4eb28a0816f2bd06cb6ba3eea4d"
server_key_version: "encryptionkey@0"
I20260925 15:16:12.066588 27195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:16:12.087832 27195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:16:12.092005 27195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:16:12.092274 27195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:16:12.110984 27195 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.62:33201
I20260925 15:16:12.111029 27261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.62:33201 every 8 connection(s)
I20260925 15:16:12.112543 27195 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
I20260925 15:16:12.115703 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 27195
I20260925 15:16:12.116509 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.1:44123
--local_ip_for_outbound_sockets=127.12.8.1
--tserver_master_addrs=127.12.8.62:33201
--webserver_port=38723
--webserver_interface=127.12.8.1
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.1.keytab
--principal=kudu/127.12.8.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:35145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:16:12.119457 27262 sys_catalog.cc:263] Verifying existing consensus state
I20260925 15:16:12.122699 27262 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 594bdcb833214f8ca0ecc6906c3c8e64: Bootstrap starting.
I20260925 15:16:12.152921 27262 log.cc:824] T 00000000000000000000000000000000 P 594bdcb833214f8ca0ecc6906c3c8e64: Log is configured to *not* fsync() on all Append() calls
I20260925 15:16:12.173712 27262 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 594bdcb833214f8ca0ecc6906c3c8e64: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 15:16:12.174441 27262 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 594bdcb833214f8ca0ecc6906c3c8e64: Bootstrap complete.
I20260925 15:16:12.189456 27262 raft_consensus.cc:348] T 00000000000000000000000000000000 P 594bdcb833214f8ca0ecc6906c3c8e64 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "594bdcb833214f8ca0ecc6906c3c8e64" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 33201 } }
I20260925 15:16:12.190699 27262 raft_consensus.cc:729] T 00000000000000000000000000000000 P 594bdcb833214f8ca0ecc6906c3c8e64 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 594bdcb833214f8ca0ecc6906c3c8e64, State: Initialized, Role: FOLLOWER
I20260925 15:16:12.191390 27262 consensus_queue.cc:260] T 00000000000000000000000000000000 P 594bdcb833214f8ca0ecc6906c3c8e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "594bdcb833214f8ca0ecc6906c3c8e64" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 33201 } }
I20260925 15:16:12.191646 27262 raft_consensus.cc:388] T 00000000000000000000000000000000 P 594bdcb833214f8ca0ecc6906c3c8e64 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:16:12.191772 27262 raft_consensus.cc:482] T 00000000000000000000000000000000 P 594bdcb833214f8ca0ecc6906c3c8e64 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:16:12.191955 27262 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 594bdcb833214f8ca0ecc6906c3c8e64 [term 2 FOLLOWER]: Advancing to term 3
I20260925 15:16:12.197359 27262 raft_consensus.cc:504] T 00000000000000000000000000000000 P 594bdcb833214f8ca0ecc6906c3c8e64 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "594bdcb833214f8ca0ecc6906c3c8e64" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 33201 } }
I20260925 15:16:12.197938 27262 leader_election.cc:302] T 00000000000000000000000000000000 P 594bdcb833214f8ca0ecc6906c3c8e64 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 594bdcb833214f8ca0ecc6906c3c8e64; no voters:
I20260925 15:16:12.198606 27262 leader_election.cc:288] T 00000000000000000000000000000000 P 594bdcb833214f8ca0ecc6906c3c8e64 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 15:16:12.198746 27266 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 594bdcb833214f8ca0ecc6906c3c8e64 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 15:16:12.199926 27266 raft_consensus.cc:686] T 00000000000000000000000000000000 P 594bdcb833214f8ca0ecc6906c3c8e64 [term 3 LEADER]: Becoming Leader. State: Replica: 594bdcb833214f8ca0ecc6906c3c8e64, State: Running, Role: LEADER
I20260925 15:16:12.200583 27262 sys_catalog.cc:564] T 00000000000000000000000000000000 P 594bdcb833214f8ca0ecc6906c3c8e64 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:16:12.200423 27266 consensus_queue.cc:237] T 00000000000000000000000000000000 P 594bdcb833214f8ca0ecc6906c3c8e64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "594bdcb833214f8ca0ecc6906c3c8e64" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 33201 } }
I20260925 15:16:12.202889 27268 sys_catalog.cc:455] T 00000000000000000000000000000000 P 594bdcb833214f8ca0ecc6906c3c8e64 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 594bdcb833214f8ca0ecc6906c3c8e64. Latest consensus state: current_term: 3 leader_uuid: "594bdcb833214f8ca0ecc6906c3c8e64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "594bdcb833214f8ca0ecc6906c3c8e64" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 33201 } } }
I20260925 15:16:12.204017 27268 sys_catalog.cc:458] T 00000000000000000000000000000000 P 594bdcb833214f8ca0ecc6906c3c8e64 [sys.catalog]: This master's current role is: LEADER
I20260925 15:16:12.203567 27267 sys_catalog.cc:455] T 00000000000000000000000000000000 P 594bdcb833214f8ca0ecc6906c3c8e64 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "594bdcb833214f8ca0ecc6906c3c8e64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "594bdcb833214f8ca0ecc6906c3c8e64" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 33201 } } }
I20260925 15:16:12.204159 27267 sys_catalog.cc:458] T 00000000000000000000000000000000 P 594bdcb833214f8ca0ecc6906c3c8e64 [sys.catalog]: This master's current role is: LEADER
I20260925 15:16:12.215351 27273 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:16:12.224193 27273 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:16:12.225785 27273 catalog_manager.cc:1259] Loaded cluster ID: d0808ddb0aa64a32a4f15a30c19c15c3
I20260925 15:16:12.225912 27273 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260925 15:16:12.227835 27273 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f238752d9ad google::LogMessage::Fail() at ??:0
@ 0x7f238753088a google::LogMessage::SendToLog() at ??:0
@ 0x7f238752d4de google::LogMessage::Flush() at ??:0
@ 0x7f238752ea19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f239cf60512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f239cf5d976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f239cfe5610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f239cfe5439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f238a179345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f238a17c91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f238a17c799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f238a157eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f238d4316db start_thread at ??:0
@ 0x7f23863c761f clone at ??:0
W20260925 15:16:12.569721 27264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:16:12.569996 27264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:16:12.570096 27264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:16:12.577256 27264 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:16:12.577365 27264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:16:12.577476 27264 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.1
I20260925 15:16:12.587157 27264 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:35145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.1.keytab
--principal=kudu/127.12.8.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.1:44123
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.8.1
--webserver_port=38723
--tserver_master_addrs=127.12.8.62:33201
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.1
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:16:12.588433 27264 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:16:12.589679 27264 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:16:12.602437 27302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:16:12.602497 27300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:16:12.602519 27299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:16:12.603264 27264 server_base.cc:1034] running on GCE node
I20260925 15:16:12.603899 27264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:16:12.604833 27264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:16:12.606048 27264 hybrid_clock.cc:648] HybridClock initialized: now 1790349372606003 us; error 28 us; skew 500 ppm
I20260925 15:16:12.606403 27264 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:16:12 dist-test-slave-p468 krb5kdc[26118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349372, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:16:12.609340 27264 init.cc:377] Logged in from keytab as kudu/127.12.8.1@KRBTEST.COM (short username kudu)
I20260925 15:16:12.610780 27264 webserver.cc:460] Webserver started at http://127.12.8.1:38723/ using document root <none> and password file <none>
I20260925 15:16:12.611390 27264 fs_manager.cc:359] Metadata directory not provided
I20260925 15:16:12.611493 27264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:16:12.616508 27264 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 15:16:12.619117 27309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:12.620448 27264 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:16:12.620623 27264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-0/wal
uuid: "e5c51475c8dd40cfb94ec868907836ca"
format_stamp: "Formatted at 2026-09-25 15:16:07 on dist-test-slave-p468"
server_key: "02203ba1aa34ee214db4ff6de2e8974d"
server_key_iv: "697a5fac1ab3e5e763f6fb0ef37ec69e"
server_key_version: "encryptionkey@0"
I20260925 15:16:12.621577 27264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:16:12.668171 27264 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:16:12.672683 27264 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:16:12.672967 27264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:16:12.673946 27264 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:16:12.675305 27264 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:16:12.675393 27264 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:12.675550 27264 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:16:12.675654 27264 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:12.709740 27264 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.1:44123
I20260925 15:16:12.709851 27421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.1:44123 every 8 connection(s)
I20260925 15:16:12.711306 27264 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-0/data/info.pb
I20260925 15:16:12.714370 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 27264
I20260925 15:16:12.715165 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.2:46647
--local_ip_for_outbound_sockets=127.12.8.2
--tserver_master_addrs=127.12.8.62:33201
--webserver_port=34027
--webserver_interface=127.12.8.2
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.2.keytab
--principal=kudu/127.12.8.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:35145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:16:12.716813 27424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:12.714395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:33201 (local address 127.12.8.1:34665)
0925 15:16:12.715310 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:12.715590 (+ 280us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:33201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":317,"threads_started":1}
W20260925 15:16:12.717579 27422 heartbeater.cc:644] Failed to heartbeat to 127.12.8.62:33201 (0 consecutive failures): Network error: Failed to ping master at 127.12.8.62:33201: Client connection negotiation failed: client connection to 127.12.8.62:33201: connect: Connection refused (error 111)
I20260925 15:16:12.718606 27424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:12.718114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:33201 (local address 127.12.8.1:40361)
0925 15:16:12.718360 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:12.718452 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:33201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260925 15:16:12.719616 27424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:12.719169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:33201 (local address 127.12.8.1:46981)
0925 15:16:12.719345 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:12.719447 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:33201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260925 15:16:12.719996 27422 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 15:16:13.041342 27425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:16:13.041605 27425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:16:13.041769 27425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:16:13.048904 27425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:16:13.049028 27425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:16:13.049211 27425 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.2
I20260925 15:16:13.058473 27425 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:35145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.2.keytab
--principal=kudu/127.12.8.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.2:46647
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.8.2
--webserver_port=34027
--tserver_master_addrs=127.12.8.62:33201
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.2
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:16:13.059670 27425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:16:13.061054 27425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:16:13.074285 27430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:16:13.074301 27431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:16:13.074378 27433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:16:13.075120 27425 server_base.cc:1034] running on GCE node
I20260925 15:16:13.075826 27425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:16:13.076740 27425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:16:13.077984 27425 hybrid_clock.cc:648] HybridClock initialized: now 1790349373077886 us; error 82 us; skew 500 ppm
I20260925 15:16:13.078415 27425 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:16:13 dist-test-slave-p468 krb5kdc[26118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349373, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:16:13.081758 27425 init.cc:377] Logged in from keytab as kudu/127.12.8.2@KRBTEST.COM (short username kudu)
I20260925 15:16:13.083140 27425 webserver.cc:460] Webserver started at http://127.12.8.2:34027/ using document root <none> and password file <none>
I20260925 15:16:13.083762 27425 fs_manager.cc:359] Metadata directory not provided
I20260925 15:16:13.083871 27425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:16:13.089346 27425 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260925 15:16:13.092269 27440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:13.093719 27425 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 15:16:13.093856 27425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-1/wal
uuid: "1b034ca722914ac1ac4b23691a36c014"
format_stamp: "Formatted at 2026-09-25 15:16:07 on dist-test-slave-p468"
server_key: "18f60b2de7d1d18d7fc5dca16b0ab3ee"
server_key_iv: "282252730ca62333142d9e7f224eafdc"
server_key_version: "encryptionkey@0"
I20260925 15:16:13.094949 27425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:16:13.122392 27425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:16:13.127215 27425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:16:13.127460 27425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:16:13.128477 27425 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:16:13.129761 27425 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:16:13.129858 27425 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:13.129940 27425 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:16:13.129999 27425 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:13.164122 27425 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.2:46647
I20260925 15:16:13.164198 27552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.2:46647 every 8 connection(s)
I20260925 15:16:13.165624 27425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-1/data/info.pb
I20260925 15:16:13.168713 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 27425
I20260925 15:16:13.169389 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.3:41841
--local_ip_for_outbound_sockets=127.12.8.3
--tserver_master_addrs=127.12.8.62:33201
--webserver_port=32787
--webserver_interface=127.12.8.3
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.3.keytab
--principal=kudu/127.12.8.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:35145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:16:13.170692 27555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:13.168617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:33201 (local address 127.12.8.2:34185)
0925 15:16:13.169486 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:13.169715 (+ 229us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:33201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":401,"threads_started":1}
W20260925 15:16:13.171329 27553 heartbeater.cc:644] Failed to heartbeat to 127.12.8.62:33201 (0 consecutive failures): Network error: Failed to ping master at 127.12.8.62:33201: Client connection negotiation failed: client connection to 127.12.8.62:33201: connect: Connection refused (error 111)
I20260925 15:16:13.172183 27555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:13.171886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:33201 (local address 127.12.8.2:35615)
0925 15:16:13.172026 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:13.172087 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:33201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260925 15:16:13.173100 27555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:13.172817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:33201 (local address 127.12.8.2:48589)
0925 15:16:13.172936 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:13.173000 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:33201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 15:16:13.173389 27553 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 15:16:13.486284 27556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:16:13.486593 27556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:16:13.486786 27556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:16:13.493969 27556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:16:13.494065 27556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:16:13.494289 27556 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.3
I20260925 15:16:13.504027 27556 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:35145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.3.keytab
--principal=kudu/127.12.8.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.3:41841
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.8.3
--webserver_port=32787
--tserver_master_addrs=127.12.8.62:33201
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.3
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:16:13.505280 27556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:16:13.506631 27556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:16:13.519083 27561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:16:13.519006 27562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:16:13.519059 27564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:16:13.520182 27556 server_base.cc:1034] running on GCE node
I20260925 15:16:13.520721 27556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:16:13.521595 27556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:16:13.522783 27556 hybrid_clock.cc:648] HybridClock initialized: now 1790349373522730 us; error 48 us; skew 500 ppm
I20260925 15:16:13.523096 27556 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:16:13 dist-test-slave-p468 krb5kdc[26118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349373, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:16:13.526387 27556 init.cc:377] Logged in from keytab as kudu/127.12.8.3@KRBTEST.COM (short username kudu)
I20260925 15:16:13.527653 27556 webserver.cc:460] Webserver started at http://127.12.8.3:32787/ using document root <none> and password file <none>
I20260925 15:16:13.528241 27556 fs_manager.cc:359] Metadata directory not provided
I20260925 15:16:13.528337 27556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:16:13.533277 27556 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 15:16:13.535758 27571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:13.537020 27556 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 15:16:13.537186 27556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-2/wal
uuid: "33c2d12159954b88ae9974464003472f"
format_stamp: "Formatted at 2026-09-25 15:16:07 on dist-test-slave-p468"
server_key: "eb96f366841014226ddfe887ce681d42"
server_key_iv: "24329e77fbeb1b69e7aaa732ed670f0c"
server_key_version: "encryptionkey@0"
I20260925 15:16:13.538105 27556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:16:13.561088 27556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:16:13.565415 27556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:16:13.565692 27556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:16:13.566740 27556 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:16:13.568087 27556 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:16:13.568181 27556 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:13.568324 27556 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:16:13.568424 27556 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:13.602367 27556 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.3:41841
I20260925 15:16:13.602458 27683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.3:41841 every 8 connection(s)
I20260925 15:16:13.603825 27556 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349162370577-12320-0/minicluster-data/ts-2/data/info.pb
I20260925 15:16:13.608783 27686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:13.606909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:33201 (local address 127.12.8.3:55763)
0925 15:16:13.607694 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:13.607905 (+ 211us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:33201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":335,"threads_started":1}
W20260925 15:16:13.609489 27684 heartbeater.cc:644] Failed to heartbeat to 127.12.8.62:33201 (0 consecutive failures): Network error: Failed to ping master at 127.12.8.62:33201: Client connection negotiation failed: client connection to 127.12.8.62:33201: connect: Connection refused (error 111)
I20260925 15:16:13.610363 27686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:13.609964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:33201 (local address 127.12.8.3:55747)
0925 15:16:13.610093 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:13.610158 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:33201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260925 15:16:13.611256 27686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:13.610905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:33201 (local address 127.12.8.3:40975)
0925 15:16:13.611057 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:13.611145 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:33201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260925 15:16:13.611517 27684 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 15:16:13.611961 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 27556
I20260925 15:16:13.721351 27688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:13.720490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:33201 (local address 127.12.8.1:50931)
0925 15:16:13.720987 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:13.721086 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:33201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":240,"threads_started":1}
I20260925 15:16:14.174850 27689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:14.173950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:33201 (local address 127.12.8.2:42997)
0925 15:16:14.174522 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:14.174632 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:33201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":299,"threads_started":1}
I20260925 15:16:14.612716 27690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:14.611994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:33201 (local address 127.12.8.3:33881)
0925 15:16:14.612411 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:14.612499 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:33201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":198,"threads_started":1}
I20260925 15:16:14.723047 27691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:14.722371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:33201 (local address 127.12.8.1:43833)
0925 15:16:14.722802 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:14.722879 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:33201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":205,"threads_started":1}
I20260925 15:16:15.176393 27692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:15.175691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:33201 (local address 127.12.8.2:60971)
0925 15:16:15.176124 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:15.176201 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:33201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":218,"threads_started":1}
I20260925 15:16:15.614408 27693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:15.613624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:33201 (local address 127.12.8.3:42371)
0925 15:16:15.614073 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:15.614162 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:33201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":223,"threads_started":1}
I20260925 15:16:15.724630 27694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:15.723893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:33201 (local address 127.12.8.1:57355)
0925 15:16:15.724326 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:15.724430 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:33201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":227,"threads_started":1}
I20260925 15:16:16.177821 27695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:16.177168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:33201 (local address 127.12.8.2:45721)
0925 15:16:16.177583 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:16.177658 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:33201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":217,"threads_started":1}
I20260925 15:16:16.615903 27696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:16.615221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:33201 (local address 127.12.8.3:44665)
0925 15:16:16.615667 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:16.615741 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:33201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":264,"threads_started":1}
I20260925 15:16:16.726341 27697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:16.725519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:33201 (local address 127.12.8.1:41443)
0925 15:16:16.726043 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:16.726123 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:33201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":312,"threads_started":1}
I20260925 15:16:17.179384 27698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:17.178687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:33201 (local address 127.12.8.2:34219)
0925 15:16:17.179138 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:17.179214 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:33201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":225,"threads_started":1}
I20260925 15:16:17.617399 27699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:17.616696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:33201 (local address 127.12.8.3:42287)
0925 15:16:17.617135 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:17.617210 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:33201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":207,"threads_started":1}
I20260925 15:16:17.727993 27700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:17.727220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:33201 (local address 127.12.8.1:51247)
0925 15:16:17.727750 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:17.727828 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:33201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":301,"threads_started":1}
I20260925 15:16:18.181160 27701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:18.180289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:33201 (local address 127.12.8.2:38699)
0925 15:16:18.180837 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:18.180951 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:33201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":290,"threads_started":1}
I20260925 15:16:18.619246 27702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:18.618468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:33201 (local address 127.12.8.3:49931)
0925 15:16:18.618935 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:18.619015 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:33201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":230,"threads_started":1}
I20260925 15:16:18.729631 27703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:18.728861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:33201 (local address 127.12.8.1:50967)
0925 15:16:18.729326 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:18.729421 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:33201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":240,"threads_started":1}
I20260925 15:16:19.182816 27704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:19.182047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:33201 (local address 127.12.8.2:53291)
0925 15:16:19.182536 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:19.182653 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:33201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":285,"threads_started":1}
I20260925 15:16:19.620935 27705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:19.620143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:33201 (local address 127.12.8.3:35667)
0925 15:16:19.620640 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:19.620777 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:33201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":271,"threads_started":1}
I20260925 15:16:19.731402 27706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:19.730578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:33201 (local address 127.12.8.1:51801)
0925 15:16:19.731097 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:19.731200 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:33201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":232,"threads_started":1}
I20260925 15:16:20.184546 27707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:20.183751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:33201 (local address 127.12.8.2:35221)
0925 15:16:20.184246 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:20.184330 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:33201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":222,"threads_started":1}
I20260925 15:16:20.622820 27708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:20.621809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:33201 (local address 127.12.8.3:37887)
0925 15:16:20.622426 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:20.622531 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:33201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":222,"threads_started":1}
I20260925 15:16:20.732930 27709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:20.732281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:33201 (local address 127.12.8.1:41945)
0925 15:16:20.732693 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:20.732768 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:33201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":215,"threads_started":1}
I20260925 15:16:21.186115 27710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:21.185439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:33201 (local address 127.12.8.2:45735)
0925 15:16:21.185876 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:21.185951 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:33201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":216,"threads_started":1}
I20260925 15:16:21.624486 27711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:21.623737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:33201 (local address 127.12.8.3:44073)
0925 15:16:21.624162 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:21.624266 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:33201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":213,"threads_started":1}
I20260925 15:16:21.734529 27712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:21.733734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:33201 (local address 127.12.8.1:58375)
0925 15:16:21.734181 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:21.734337 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:33201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":223,"threads_started":1}
I20260925 15:16:22.187806 27713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:22.187024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:33201 (local address 127.12.8.2:48649)
0925 15:16:22.187558 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:22.187633 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:33201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":297,"threads_started":1}
I20260925 15:16:22.626466 27714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:22.625479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:33201 (local address 127.12.8.3:56833)
0925 15:16:22.626140 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:22.626278 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:33201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":438,"threads_started":1}
I20260925 15:16:22.736120 27715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:22.735436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:33201 (local address 127.12.8.1:37011)
0925 15:16:22.735886 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:22.735962 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:33201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":248,"threads_started":1}
I20260925 15:16:23.189453 27716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:23.188768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:33201 (local address 127.12.8.2:52351)
0925 15:16:23.189215 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:23.189290 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:33201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":253,"threads_started":1}
I20260925 15:16:23.628022 27717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:23.627355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:33201 (local address 127.12.8.3:49445)
0925 15:16:23.627790 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:23.627866 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:33201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":234,"threads_started":1}
I20260925 15:16:23.737744 27718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:23.737021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:33201 (local address 127.12.8.1:38295)
0925 15:16:23.737483 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:23.737559 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:33201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":241,"threads_started":1}
I20260925 15:16:24.191139 27719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:24.190435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:33201 (local address 127.12.8.2:52645)
0925 15:16:24.190880 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:24.190963 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:33201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":233,"threads_started":1}
I20260925 15:16:24.629565 27720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:24.628913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:33201 (local address 127.12.8.3:52157)
0925 15:16:24.629310 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:24.629392 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:33201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":200,"threads_started":1}
I20260925 15:16:24.739388 27721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:24.738692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:33201 (local address 127.12.8.1:53597)
0925 15:16:24.739123 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:24.739220 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:33201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":220,"threads_started":1}
I20260925 15:16:25.192687 27722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:25.192004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:33201 (local address 127.12.8.2:52705)
0925 15:16:25.192431 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:25.192518 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:33201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":213,"threads_started":1}
I20260925 15:16:25.631199 27723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:25.630474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:33201 (local address 127.12.8.3:50229)
0925 15:16:25.630943 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:25.631037 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:33201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":233,"threads_started":1}
I20260925 15:16:25.741145 27724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:25.740305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:33201 (local address 127.12.8.1:50149)
0925 15:16:25.740779 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:25.740866 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:33201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":231,"threads_started":1}
I20260925 15:16:26.194403 27725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:26.193557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:33201 (local address 127.12.8.2:33049)
0925 15:16:26.194065 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:26.194143 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:33201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":239,"threads_started":1}
I20260925 15:16:26.632858 27726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:26.632066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:33201 (local address 127.12.8.3:54145)
0925 15:16:26.632552 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:26.632628 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:33201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":235,"threads_started":1}
I20260925 15:16:26.743067 27727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:26.742093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:33201 (local address 127.12.8.1:54921)
0925 15:16:26.742680 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:26.742813 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:33201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":295,"threads_started":1}
I20260925 15:16:27.196074 27728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:27.195252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:33201 (local address 127.12.8.2:55745)
0925 15:16:27.195745 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:27.195862 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:33201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":266,"threads_started":1}
I20260925 15:16:27.634466 27729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:27.633741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:33201 (local address 127.12.8.3:39043)
0925 15:16:27.634154 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:27.634283 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:33201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":213,"threads_started":1}
I20260925 15:16:27.744885 27730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:27.744149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:33201 (local address 127.12.8.1:38719)
0925 15:16:27.744623 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:27.744713 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:33201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":277,"threads_started":1}
I20260925 15:16:28.197906 27731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:28.197150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:33201 (local address 127.12.8.2:51323)
0925 15:16:28.197643 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:28.197728 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:33201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":290,"threads_started":1}
I20260925 15:16:28.623237 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 27264
I20260925 15:16:28.637091 27733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:28.635525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:33201 (local address 127.12.8.3:60253)
0925 15:16:28.636706 (+ 1181us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:28.636823 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:33201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1010,"thread_start_us":882,"threads_started":1}
I20260925 15:16:28.643571 27732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:28.624338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52720 (local address 127.12.8.1:44123)
0925 15:16:28.625009 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:28.625024 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:28.625063 (+ 39us) server_negotiation.cc:407] Connection header received
0925 15:16:28.627233 (+ 2170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:28.627244 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:28.627695 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:28.628001 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:28.629396 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:28.630804 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:28.632123 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:28.632801 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:28.632947 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:28.635719 (+ 2772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:16:28.635727 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:16:28.635742 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:16:28.635789 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:16:28.639173 (+ 3384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:28.640058 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:28.640067 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:28.640076 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:28.640274 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:28.640750 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:28.640755 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:28.640758 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:28.641411 (+ 653us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:16:28.642426 (+ 1015us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:28.642751 (+ 325us) server_negotiation.cc:299] Negotiation successful
0925 15:16:28.643178 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":326,"threads_started":1}
I20260925 15:16:28.645951 27417 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:16:28.731406 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 27264
I20260925 15:16:28.748392 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 27425
I20260925 15:16:28.766579 27735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:28.749488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40990 (local address 127.12.8.2:46647)
0925 15:16:28.750015 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:28.750029 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:28.750065 (+ 36us) server_negotiation.cc:407] Connection header received
0925 15:16:28.751694 (+ 1629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:28.751703 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:28.752134 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:28.752487 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:28.753962 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:28.755248 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:28.756539 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:28.757209 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:28.757404 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:28.760054 (+ 2650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:16:28.760062 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:16:28.760076 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:16:28.760113 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:16:28.763223 (+ 3110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:28.763941 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:28.763950 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:28.763956 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:28.764147 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:28.764538 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:28.764544 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:28.764547 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:28.765043 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:16:28.765349 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:28.765813 (+ 464us) server_negotiation.cc:299] Negotiation successful
0925 15:16:28.766287 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":260,"threads_started":1}
I20260925 15:16:28.768013 27548 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:16:28.852973 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 27425
I20260925 15:16:28.870249 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 27556
I20260925 15:16:28.887785 27737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:28.871291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52182 (local address 127.12.8.3:41841)
0925 15:16:28.871940 (+ 649us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:28.871953 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:28.871993 (+ 40us) server_negotiation.cc:407] Connection header received
0925 15:16:28.873645 (+ 1652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:28.873654 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:28.874083 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:28.874446 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:28.875793 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:28.877045 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:28.878337 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:28.879106 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:28.879248 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:28.881782 (+ 2534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:16:28.881789 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:16:28.881802 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:16:28.881835 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:16:28.884838 (+ 3003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:28.885447 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:28.885453 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:28.885459 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:28.885624 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:28.886053 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:28.886056 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:28.886058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:28.886616 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:16:28.886838 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:28.887190 (+ 352us) server_negotiation.cc:299] Negotiation successful
0925 15:16:28.887492 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":303,"threads_started":1}
I20260925 15:16:28.889201 27679 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:16:28.973150 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 27556
2026-09-25T15:16:28Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24248 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:16:29 dist-test-slave-p468 krb5kdc[27741](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:16:29 dist-test-slave-p468 krb5kdc[27741](info): set up 2 sockets
Sep 25 15:16:29 dist-test-slave-p468 krb5kdc[27741](info): commencing operation
krb5kdc: starting...
W20260925 15:16:30.082924 12320 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.002s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/krb5kdc/test-user.keytab.
Sep 25 15:16:30 dist-test-slave-p468 krb5kdc[27741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:16:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:16:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.8.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab.
Entry for principal kudu/127.12.8.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab.
I20260925 15:16:30.323063 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.62:36947
--webserver_interface=127.12.8.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab
--principal=kudu/127.12.8.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:38153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.62:36947
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:16:30.650080 27754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:16:30.650435 27754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:16:30.650532 27754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:16:30.657827 27754 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 15:16:30.657923 27754 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 15:16:30.657961 27754 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 15:16:30.658025 27754 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:16:30.658078 27754 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:16:30.658138 27754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:16:30.658182 27754 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:16:30.658304 27754 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:16:30.668046 27754 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:38153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.12.8.62:36947
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab
--principal=kudu/127.12.8.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.62:36947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.8.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:16:30.669219 27754 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:16:30.670519 27754 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:16:30.682459 27759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:16:30.682480 27762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:16:30.682389 27760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:16:30.683732 27754 server_base.cc:1034] running on GCE node
I20260925 15:16:30.684556 27754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:16:30.685904 27754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:16:30.687124 27754 hybrid_clock.cc:648] HybridClock initialized: now 1790349390687059 us; error 36 us; skew 500 ppm
I20260925 15:16:30.687463 27754 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:16:30 dist-test-slave-p468 krb5kdc[27741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349390, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:16:30.696532 27754 init.cc:377] Logged in from keytab as kudu/127.12.8.62@KRBTEST.COM (short username kudu)
I20260925 15:16:30.698086 27754 webserver.cc:460] Webserver started at http://127.12.8.62:41499/ using document root <none> and password file <none>
I20260925 15:16:30.698815 27754 fs_manager.cc:359] Metadata directory not provided
I20260925 15:16:30.698927 27754 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:16:30.699292 27754 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:16:30.702291 27754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/master-0/data/instance:
uuid: "46041bb901664061b44eb699831c4e55"
format_stamp: "Formatted at 2026-09-25 15:16:30 on dist-test-slave-p468"
server_key: "d1172c8c8c128130cd60dc56711820cc"
server_key_iv: "0884fac03bac734537c56062624c1806"
server_key_version: "encryptionkey@0"
I20260925 15:16:30.703083 27754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/master-0/wal/instance:
uuid: "46041bb901664061b44eb699831c4e55"
format_stamp: "Formatted at 2026-09-25 15:16:30 on dist-test-slave-p468"
server_key: "d1172c8c8c128130cd60dc56711820cc"
server_key_iv: "0884fac03bac734537c56062624c1806"
server_key_version: "encryptionkey@0"
I20260925 15:16:30.708112 27754 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 15:16:30.711697 27769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:30.713071 27754 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 15:16:30.713258 27754 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/master-0/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/master-0/wal
uuid: "46041bb901664061b44eb699831c4e55"
format_stamp: "Formatted at 2026-09-25 15:16:30 on dist-test-slave-p468"
server_key: "d1172c8c8c128130cd60dc56711820cc"
server_key_iv: "0884fac03bac734537c56062624c1806"
server_key_version: "encryptionkey@0"
I20260925 15:16:30.713485 27754 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:16:30.745910 27754 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:16:30.750397 27754 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:16:30.750771 27754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:16:30.769730 27754 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.62:36947
I20260925 15:16:30.769754 27820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.62:36947 every 8 connection(s)
I20260925 15:16:30.771113 27754 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
I20260925 15:16:30.774860 27821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:16:30.776063 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 27754
I20260925 15:16:30.776281 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/master-0/wal/instance
I20260925 15:16:30.777149 12320 external_mini_cluster.cc:1428] Setting key fb3d06a6a638ab1ae74af67c5b320ae6
I20260925 15:16:30.789552 27821 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46041bb901664061b44eb699831c4e55: Bootstrap starting.
Sep 25 15:16:30 dist-test-slave-p468 krb5kdc[27741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
I20260925 15:16:30.793073 27821 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 46041bb901664061b44eb699831c4e55: Neither blocks nor log segments found. Creating new log.
I20260925 15:16:30.794137 27821 log.cc:824] T 00000000000000000000000000000000 P 46041bb901664061b44eb699831c4e55: Log is configured to *not* fsync() on all Append() calls
I20260925 15:16:30.796845 27821 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46041bb901664061b44eb699831c4e55: No bootstrap required, opened a new log
I20260925 15:16:30.800218 27824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:30.779218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34802 (local address 127.12.8.62:36947)
0925 15:16:30.779992 (+ 774us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:30.780003 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:30.780044 (+ 41us) server_negotiation.cc:407] Connection header received
0925 15:16:30.781608 (+ 1564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:30.781627 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:30.781997 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:30.782380 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:30.784095 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:30.785338 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:30.787353 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:30.788015 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:30.788142 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:30.792883 (+ 4741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:16:30.792890 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:16:30.792904 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:16:30.792937 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:16:30.796170 (+ 3233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:30.796853 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:30.796859 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:30.796865 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:30.797088 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:30.797500 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:30.797505 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:30.797509 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:30.798029 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:16:30.798374 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:30.798855 (+ 481us) server_negotiation.cc:299] Negotiation successful
0925 15:16:30.799252 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":380,"threads_started":1}
I20260925 15:16:30.804795 27821 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46041bb901664061b44eb699831c4e55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46041bb901664061b44eb699831c4e55" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 36947 } }
I20260925 15:16:30.805195 27821 raft_consensus.cc:374] T 00000000000000000000000000000000 P 46041bb901664061b44eb699831c4e55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:16:30.805305 27821 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46041bb901664061b44eb699831c4e55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46041bb901664061b44eb699831c4e55, State: Initialized, Role: FOLLOWER
I20260925 15:16:30.805771 27821 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46041bb901664061b44eb699831c4e55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46041bb901664061b44eb699831c4e55" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 36947 } }
I20260925 15:16:30.805974 27821 raft_consensus.cc:388] T 00000000000000000000000000000000 P 46041bb901664061b44eb699831c4e55 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:16:30.806075 27821 raft_consensus.cc:482] T 00000000000000000000000000000000 P 46041bb901664061b44eb699831c4e55 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:16:30.806246 27821 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46041bb901664061b44eb699831c4e55 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:16:30.808344 27821 raft_consensus.cc:504] T 00000000000000000000000000000000 P 46041bb901664061b44eb699831c4e55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46041bb901664061b44eb699831c4e55" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 36947 } }
I20260925 15:16:30.808769 27821 leader_election.cc:302] T 00000000000000000000000000000000 P 46041bb901664061b44eb699831c4e55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 46041bb901664061b44eb699831c4e55; no voters:
I20260925 15:16:30.809307 27821 leader_election.cc:288] T 00000000000000000000000000000000 P 46041bb901664061b44eb699831c4e55 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:16:30.809473 27826 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 46041bb901664061b44eb699831c4e55 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:16:30.810285 27826 raft_consensus.cc:686] T 00000000000000000000000000000000 P 46041bb901664061b44eb699831c4e55 [term 1 LEADER]: Becoming Leader. State: Replica: 46041bb901664061b44eb699831c4e55, State: Running, Role: LEADER
I20260925 15:16:30.810842 27826 consensus_queue.cc:237] T 00000000000000000000000000000000 P 46041bb901664061b44eb699831c4e55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46041bb901664061b44eb699831c4e55" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 36947 } }
I20260925 15:16:30.811726 27821 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46041bb901664061b44eb699831c4e55 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:16:30.814514 27827 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46041bb901664061b44eb699831c4e55 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "46041bb901664061b44eb699831c4e55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46041bb901664061b44eb699831c4e55" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 36947 } } }
I20260925 15:16:30.814558 27828 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46041bb901664061b44eb699831c4e55 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46041bb901664061b44eb699831c4e55. Latest consensus state: current_term: 1 leader_uuid: "46041bb901664061b44eb699831c4e55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46041bb901664061b44eb699831c4e55" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 36947 } } }
I20260925 15:16:30.814886 27827 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46041bb901664061b44eb699831c4e55 [sys.catalog]: This master's current role is: LEADER
I20260925 15:16:30.814970 27828 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46041bb901664061b44eb699831c4e55 [sys.catalog]: This master's current role is: LEADER
I20260925 15:16:30.821344 27833 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:16:30.826558 27833 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:16:30.835664 27833 catalog_manager.cc:1347] Generated new cluster ID: 1a3299685a8e4f32a0950e03622ecab5
I20260925 15:16:30.835850 27833 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:16:30.847244 27833 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:16:30.849387 27833 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:16:30.860386 27833 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 46041bb901664061b44eb699831c4e55: Generated new TSK 0
I20260925 15:16:30.861254 27833 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.8.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.1.keytab.
Entry for principal kudu/127.12.8.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.1.keytab.
I20260925 15:16:30.922842 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.1:0
--local_ip_for_outbound_sockets=127.12.8.1
--webserver_interface=127.12.8.1
--webserver_port=0
--tserver_master_addrs=127.12.8.62:36947
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.1.keytab
--principal=kudu/127.12.8.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:38153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:16:31.245689 27847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:16:31.245939 27847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:16:31.246107 27847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:16:31.253190 27847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:16:31.253286 27847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:16:31.253446 27847 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.1
I20260925 15:16:31.262882 27847 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:38153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.1.keytab
--principal=kudu/127.12.8.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.8.1
--webserver_port=0
--tserver_master_addrs=127.12.8.62:36947
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.1
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:16:31.264060 27847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:16:31.265331 27847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:16:31.278301 27852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:16:31.278453 27853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:16:31.278218 27855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:16:31.279508 27847 server_base.cc:1034] running on GCE node
I20260925 15:16:31.280129 27847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:16:31.281011 27847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:16:31.282281 27847 hybrid_clock.cc:648] HybridClock initialized: now 1790349391282221 us; error 60 us; skew 500 ppm
I20260925 15:16:31.282608 27847 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:16:31 dist-test-slave-p468 krb5kdc[27741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349391, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:16:31.285769 27847 init.cc:377] Logged in from keytab as kudu/127.12.8.1@KRBTEST.COM (short username kudu)
I20260925 15:16:31.287101 27847 webserver.cc:460] Webserver started at http://127.12.8.1:45247/ using document root <none> and password file <none>
I20260925 15:16:31.287699 27847 fs_manager.cc:359] Metadata directory not provided
I20260925 15:16:31.287796 27847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:16:31.288029 27847 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:16:31.290797 27847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-0/data/instance:
uuid: "232b8a44ea8e4bd39da1eed7fcb322e4"
format_stamp: "Formatted at 2026-09-25 15:16:31 on dist-test-slave-p468"
server_key: "3c2680983affd0ece3432972c9ed0d94"
server_key_iv: "7968c95a7ada86ad22f2d74947990827"
server_key_version: "encryptionkey@0"
I20260925 15:16:31.291486 27847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-0/wal/instance:
uuid: "232b8a44ea8e4bd39da1eed7fcb322e4"
format_stamp: "Formatted at 2026-09-25 15:16:31 on dist-test-slave-p468"
server_key: "3c2680983affd0ece3432972c9ed0d94"
server_key_iv: "7968c95a7ada86ad22f2d74947990827"
server_key_version: "encryptionkey@0"
I20260925 15:16:31.296305 27847 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 15:16:31.299552 27862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:31.300817 27847 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 15:16:31.300935 27847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-0/wal
uuid: "232b8a44ea8e4bd39da1eed7fcb322e4"
format_stamp: "Formatted at 2026-09-25 15:16:31 on dist-test-slave-p468"
server_key: "3c2680983affd0ece3432972c9ed0d94"
server_key_iv: "7968c95a7ada86ad22f2d74947990827"
server_key_version: "encryptionkey@0"
I20260925 15:16:31.301100 27847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:16:31.338281 27847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:16:31.342700 27847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:16:31.342936 27847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:16:31.343824 27847 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:16:31.345139 27847 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:16:31.345232 27847 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:31.345314 27847 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:16:31.345358 27847 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:31.379029 27847 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.1:43327
I20260925 15:16:31.379148 27974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.1:43327 every 8 connection(s)
I20260925 15:16:31.380300 27847 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-0/data/info.pb
I20260925 15:16:31.386132 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 27847
I20260925 15:16:31.386375 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-0/wal/instance
I20260925 15:16:31.387203 12320 external_mini_cluster.cc:1428] Setting key 160caab210d5fac6c9690358e3c727be
Sep 25 15:16:31 dist-test-slave-p468 krb5kdc[27741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349391, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.1@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
I20260925 15:16:31.401012 27977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:31.382897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.1:39713 (local address 127.12.8.62:36947)
0925 15:16:31.383389 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:31.383397 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:31.384333 (+ 936us) server_negotiation.cc:407] Connection header received
0925 15:16:31.386293 (+ 1960us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:31.386298 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:31.386383 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:31.386514 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:31.388427 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:31.389281 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:31.390927 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:31.391325 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:31.391464 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:31.394929 (+ 3465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:16:31.394935 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:16:31.394939 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:16:31.394969 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:16:31.397987 (+ 3018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:31.398755 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:31.398760 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:31.398762 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:31.398881 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:31.399318 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:31.399323 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:31.399328 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:31.399588 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:16:31.399740 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:31.400428 (+ 688us) server_negotiation.cc:299] Negotiation successful
0925 15:16:31.400657 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":254,"threads_started":1}
I20260925 15:16:31.401930 27978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:31.383463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:36947 (local address 127.12.8.1:39713)
0925 15:16:31.384161 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:31.384211 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:31.385958 (+ 1747us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:31.386813 (+ 855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:31.386825 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:31.387606 (+ 781us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:16:31.388244 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:31.388262 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:31.389449 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:31.389454 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:31.390778 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:31.390790 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:31.391556 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:31.391564 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:31.391799 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:31.392487 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:16:31.392508 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:16:31.394709 (+ 2201us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:16:31.398129 (+ 3420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:16:31.398140 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:16:31.398158 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:16:31.398617 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:16:31.398992 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:16:31.398997 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:16:31.399001 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:16:31.399198 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:16:31.399725 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:16:31.399733 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:16:31.400052 (+ 319us) client_negotiation.cc:769] Sending connection context
0925 15:16:31.400318 (+ 266us) client_negotiation.cc:240] Negotiation successful
0925 15:16:31.400822 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":281,"threads_started":1}
I20260925 15:16:31.403571 27975 heartbeater.cc:347] Connected to a master server at 127.12.8.62:36947
I20260925 15:16:31.403923 27975 heartbeater.cc:464] Registering TS with master...
I20260925 15:16:31.404721 27975 heartbeater.cc:511] Master 127.12.8.62:36947 requested a full tablet report, sending...
I20260925 15:16:31.406860 27786 ts_manager.cc:194] Registered new tserver with Master: 232b8a44ea8e4bd39da1eed7fcb322e4 (127.12.8.1:43327)
I20260925 15:16:31.408275 27786 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.1@KRBTEST.COM'} at 127.12.8.1:39713
WARNING: no policy specified for kudu/127.12.8.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.2.keytab.
Entry for principal kudu/127.12.8.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.2.keytab.
I20260925 15:16:31.440833 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.2:0
--local_ip_for_outbound_sockets=127.12.8.2
--webserver_interface=127.12.8.2
--webserver_port=0
--tserver_master_addrs=127.12.8.62:36947
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.2.keytab
--principal=kudu/127.12.8.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:38153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:16:31.754952 27981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:16:31.755211 27981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:16:31.755398 27981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:16:31.762463 27981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:16:31.762588 27981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:16:31.762776 27981 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.2
I20260925 15:16:31.772135 27981 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:38153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.2.keytab
--principal=kudu/127.12.8.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.8.2
--webserver_port=0
--tserver_master_addrs=127.12.8.62:36947
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.2
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:16:31.773356 27981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:16:31.774650 27981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:16:31.787379 27986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:16:31.787305 27987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:16:31.787353 27989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:16:31.788215 27981 server_base.cc:1034] running on GCE node
I20260925 15:16:31.788724 27981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:16:31.789633 27981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:16:31.790872 27981 hybrid_clock.cc:648] HybridClock initialized: now 1790349391790840 us; error 53 us; skew 500 ppm
I20260925 15:16:31.791204 27981 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:16:31 dist-test-slave-p468 krb5kdc[27741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349391, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:16:31.794418 27981 init.cc:377] Logged in from keytab as kudu/127.12.8.2@KRBTEST.COM (short username kudu)
I20260925 15:16:31.795771 27981 webserver.cc:460] Webserver started at http://127.12.8.2:36279/ using document root <none> and password file <none>
I20260925 15:16:31.796429 27981 fs_manager.cc:359] Metadata directory not provided
I20260925 15:16:31.796573 27981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:16:31.796865 27981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:16:31.799691 27981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-1/data/instance:
uuid: "7bba3fc3f1e84d7c8254a77f05fd82df"
format_stamp: "Formatted at 2026-09-25 15:16:31 on dist-test-slave-p468"
server_key: "841dbc33588ead37eb9b219a185bc313"
server_key_iv: "df420a438caa3a76f6bfff89b1e9ac9a"
server_key_version: "encryptionkey@0"
I20260925 15:16:31.800420 27981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-1/wal/instance:
uuid: "7bba3fc3f1e84d7c8254a77f05fd82df"
format_stamp: "Formatted at 2026-09-25 15:16:31 on dist-test-slave-p468"
server_key: "841dbc33588ead37eb9b219a185bc313"
server_key_iv: "df420a438caa3a76f6bfff89b1e9ac9a"
server_key_version: "encryptionkey@0"
I20260925 15:16:31.805506 27981 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 15:16:31.808888 27996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:31.810130 27981 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 15:16:31.810343 27981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-1/wal
uuid: "7bba3fc3f1e84d7c8254a77f05fd82df"
format_stamp: "Formatted at 2026-09-25 15:16:31 on dist-test-slave-p468"
server_key: "841dbc33588ead37eb9b219a185bc313"
server_key_iv: "df420a438caa3a76f6bfff89b1e9ac9a"
server_key_version: "encryptionkey@0"
I20260925 15:16:31.810550 27981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:16:31.831764 27981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:16:31.835634 27981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:16:31.835918 27981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:16:31.836824 27981 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:16:31.838071 27981 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:16:31.838193 27981 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:31.838363 27981 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:16:31.838438 27981 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:31.875523 27981 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.2:45177
I20260925 15:16:31.875581 28108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.2:45177 every 8 connection(s)
I20260925 15:16:31.876816 27981 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-1/data/info.pb
I20260925 15:16:31.882146 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 27981
I20260925 15:16:31.882360 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-1/wal/instance
I20260925 15:16:31.882926 12320 external_mini_cluster.cc:1428] Setting key ae37961972a4871dc1b10bb03271e939
Sep 25 15:16:31 dist-test-slave-p468 krb5kdc[27741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349391, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.2@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
I20260925 15:16:31.899302 27977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:31.879393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.2:39813 (local address 127.12.8.62:36947)
0925 15:16:31.879578 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:31.879582 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:31.880728 (+ 1146us) server_negotiation.cc:407] Connection header received
0925 15:16:31.883010 (+ 2282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:31.883019 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:31.883111 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:31.883192 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:31.884990 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:31.885752 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:31.888297 (+ 2545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:31.888742 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:31.888843 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:31.892976 (+ 4133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:16:31.892981 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:16:31.892985 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:16:31.893016 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:16:31.896174 (+ 3158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:31.896901 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:31.896912 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:31.896914 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:31.897048 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:31.897503 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:31.897512 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:31.897514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:31.897779 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:16:31.897937 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:31.898728 (+ 791us) server_negotiation.cc:299] Negotiation successful
0925 15:16:31.898956 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 15:16:31.900280 28111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:31.879876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:36947 (local address 127.12.8.2:39813)
0925 15:16:31.880568 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:31.880606 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:31.882679 (+ 2073us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:31.883471 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:31.883482 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:31.884086 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:16:31.884832 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:31.884851 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:31.885900 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:31.885907 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:31.888108 (+ 2201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:31.888127 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:31.888972 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:31.888981 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:31.889307 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:31.890036 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:16:31.890075 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:16:31.892748 (+ 2673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:16:31.896324 (+ 3576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:16:31.896334 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:16:31.896352 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:16:31.896767 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:16:31.897191 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:16:31.897197 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:16:31.897201 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:16:31.897380 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:16:31.897912 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:16:31.897919 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:16:31.898334 (+ 415us) client_negotiation.cc:769] Sending connection context
0925 15:16:31.898612 (+ 278us) client_negotiation.cc:240] Negotiation successful
0925 15:16:31.899145 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":279,"threads_started":1}
I20260925 15:16:31.901790 28109 heartbeater.cc:347] Connected to a master server at 127.12.8.62:36947
I20260925 15:16:31.902135 28109 heartbeater.cc:464] Registering TS with master...
I20260925 15:16:31.903216 28109 heartbeater.cc:511] Master 127.12.8.62:36947 requested a full tablet report, sending...
I20260925 15:16:31.905007 27786 ts_manager.cc:194] Registered new tserver with Master: 7bba3fc3f1e84d7c8254a77f05fd82df (127.12.8.2:45177)
I20260925 15:16:31.906404 27786 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.2@KRBTEST.COM'} at 127.12.8.2:39813
WARNING: no policy specified for kudu/127.12.8.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.3.keytab.
Entry for principal kudu/127.12.8.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.3.keytab.
I20260925 15:16:31.941469 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.3:0
--local_ip_for_outbound_sockets=127.12.8.3
--webserver_interface=127.12.8.3
--webserver_port=0
--tserver_master_addrs=127.12.8.62:36947
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.3.keytab
--principal=kudu/127.12.8.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:38153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:16:32.261123 28114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:16:32.261431 28114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:16:32.261610 28114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:16:32.268867 28114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:16:32.269025 28114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:16:32.269208 28114 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.3
I20260925 15:16:32.278661 28114 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:38153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.3.keytab
--principal=kudu/127.12.8.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.8.3
--webserver_port=0
--tserver_master_addrs=127.12.8.62:36947
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.3
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:16:32.280072 28114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:16:32.281347 28114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:16:32.294229 28120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:16:32.294363 28122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:16:32.294312 28119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:16:32.294987 28114 server_base.cc:1034] running on GCE node
I20260925 15:16:32.295625 28114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:16:32.296706 28114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:16:32.297951 28114 hybrid_clock.cc:648] HybridClock initialized: now 1790349392297903 us; error 73 us; skew 500 ppm
I20260925 15:16:32.298339 28114 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:16:32 dist-test-slave-p468 krb5kdc[27741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349392, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:16:32.301710 28114 init.cc:377] Logged in from keytab as kudu/127.12.8.3@KRBTEST.COM (short username kudu)
I20260925 15:16:32.303176 28114 webserver.cc:460] Webserver started at http://127.12.8.3:34497/ using document root <none> and password file <none>
I20260925 15:16:32.303815 28114 fs_manager.cc:359] Metadata directory not provided
I20260925 15:16:32.303913 28114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:16:32.304150 28114 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:16:32.307545 28114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-2/data/instance:
uuid: "a2cf2ac0a7a14b00986fab5569735fb8"
format_stamp: "Formatted at 2026-09-25 15:16:32 on dist-test-slave-p468"
server_key: "49a90e388af98b837903ea5dd77191a9"
server_key_iv: "3f16fb4f69de85b6124e5ac0e382c389"
server_key_version: "encryptionkey@0"
I20260925 15:16:32.308323 28114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-2/wal/instance:
uuid: "a2cf2ac0a7a14b00986fab5569735fb8"
format_stamp: "Formatted at 2026-09-25 15:16:32 on dist-test-slave-p468"
server_key: "49a90e388af98b837903ea5dd77191a9"
server_key_iv: "3f16fb4f69de85b6124e5ac0e382c389"
server_key_version: "encryptionkey@0"
I20260925 15:16:32.313681 28114 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 15:16:32.317420 28129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:32.318876 28114 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:16:32.319051 28114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-2/wal
uuid: "a2cf2ac0a7a14b00986fab5569735fb8"
format_stamp: "Formatted at 2026-09-25 15:16:32 on dist-test-slave-p468"
server_key: "49a90e388af98b837903ea5dd77191a9"
server_key_iv: "3f16fb4f69de85b6124e5ac0e382c389"
server_key_version: "encryptionkey@0"
I20260925 15:16:32.319265 28114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:16:32.347436 28114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:16:32.351613 28114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:16:32.351934 28114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:16:32.352943 28114 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:16:32.354398 28114 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:16:32.354497 28114 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:32.354653 28114 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:16:32.354729 28114 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:32.389904 28114 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.3:46683
I20260925 15:16:32.389994 28241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.3:46683 every 8 connection(s)
I20260925 15:16:32.391215 28114 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-2/data/info.pb
I20260925 15:16:32.393420 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 28114
I20260925 15:16:32.393663 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-2/wal/instance
I20260925 15:16:32.394168 12320 external_mini_cluster.cc:1428] Setting key 63832412a0d3a1a95329c077fd5bbb83
Sep 25 15:16:32 dist-test-slave-p468 krb5kdc[27741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349392, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.3@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
I20260925 15:16:32.411049 27975 heartbeater.cc:503] Master 127.12.8.62:36947 was elected leader, sending a full tablet report...
I20260925 15:16:32.412001 27977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:32.393791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.3:55335 (local address 127.12.8.62:36947)
0925 15:16:32.394009 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:32.394016 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:32.395338 (+ 1322us) server_negotiation.cc:407] Connection header received
0925 15:16:32.397320 (+ 1982us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:32.397323 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:32.397390 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:32.397470 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:32.399004 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:32.399759 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:32.401326 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:32.401738 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:32.401831 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:32.405664 (+ 3833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:16:32.405668 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:16:32.405671 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:16:32.405704 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:16:32.408862 (+ 3158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:32.409579 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:32.409587 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:32.409590 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:32.409725 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:32.410203 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:32.410209 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:32.410212 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:32.410479 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:16:32.410640 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:32.411353 (+ 713us) server_negotiation.cc:299] Negotiation successful
0925 15:16:32.411615 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 15:16:32.413089 28244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:32.394365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:36947 (local address 127.12.8.3:55335)
0925 15:16:32.395187 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:32.395237 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:32.397075 (+ 1838us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:32.397703 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:32.397711 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:32.398292 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:16:32.398849 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:32.398865 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:32.399908 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:32.399912 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:32.401159 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:32.401176 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:32.401931 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:32.401936 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:32.402143 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:32.402958 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:16:32.402980 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:16:32.405474 (+ 2494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:16:32.408988 (+ 3514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:16:32.408994 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:16:32.409005 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:16:32.409437 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:16:32.409868 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:16:32.409873 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:16:32.409876 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:16:32.410068 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:16:32.410615 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:16:32.410623 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:16:32.410939 (+ 316us) client_negotiation.cc:769] Sending connection context
0925 15:16:32.411198 (+ 259us) client_negotiation.cc:240] Negotiation successful
0925 15:16:32.411719 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":350,"threads_started":1}
I20260925 15:16:32.414505 28242 heartbeater.cc:347] Connected to a master server at 127.12.8.62:36947
I20260925 15:16:32.414845 28242 heartbeater.cc:464] Registering TS with master...
I20260925 15:16:32.415652 28242 heartbeater.cc:511] Master 127.12.8.62:36947 requested a full tablet report, sending...
I20260925 15:16:32.417150 27786 ts_manager.cc:194] Registered new tserver with Master: a2cf2ac0a7a14b00986fab5569735fb8 (127.12.8.3:46683)
I20260925 15:16:32.418008 27786 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.3@KRBTEST.COM'} at 127.12.8.3:55335
I20260925 15:16:32.421743 12320 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:16:32.440147 27977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:32.426896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34808 (local address 127.12.8.62:36947)
0925 15:16:32.427065 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:32.427070 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:32.427483 (+ 413us) server_negotiation.cc:407] Connection header received
0925 15:16:32.427613 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:32.427617 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:32.427678 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:32.427761 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:32.429378 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:32.430099 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:32.431497 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:32.431897 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:32.431993 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:32.434573 (+ 2580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:16:32.434578 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:16:32.434581 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:16:32.434611 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:16:32.437417 (+ 2806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:32.438149 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:32.438153 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:32.438155 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:32.438431 (+ 276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:32.438838 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:32.438842 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:32.438844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:32.439197 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:16:32.439384 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:32.439751 (+ 367us) server_negotiation.cc:299] Negotiation successful
0925 15:16:32.439918 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 15:16:32.444298 27786 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34808:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:16:32.451676 27786 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:16:32.485069 28256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:32.463803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37070 (local address 127.12.8.2:45177)
0925 15:16:32.474086 (+ 10283us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:32.474097 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:32.475527 (+ 1430us) server_negotiation.cc:407] Connection header received
0925 15:16:32.475645 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:32.475654 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:32.475936 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:32.477796 (+ 1860us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:16:32.478017 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:32.479662 (+ 1645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:32.482508 (+ 2846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:32.483834 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:32.484291 (+ 457us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:32.484470 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:32.484609 (+ 139us) server_negotiation.cc:299] Negotiation successful
0925 15:16:32.484764 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10087,"thread_start_us":9982,"threads_started":1}
I20260925 15:16:32.485258 28254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:32.463609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.2:45177 (local address 127.0.0.1:37070)
0925 15:16:32.475053 (+ 11444us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:32.475090 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:32.475275 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:32.476202 (+ 927us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:32.476212 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:32.476246 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:16:32.477704 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:32.477723 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:32.479815 (+ 2092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:32.479820 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:32.482345 (+ 2525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:32.482360 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:32.484059 (+ 1699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:32.484066 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:32.484360 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:32.484416 (+ 56us) client_negotiation.cc:769] Sending connection context
0925 15:16:32.484555 (+ 139us) client_negotiation.cc:240] Negotiation successful
0925 15:16:32.484737 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11209,"mutex_wait_us":30,"thread_start_us":11059,"threads_started":1}
I20260925 15:16:32.485301 28259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:32.476490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36362 (local address 127.12.8.3:46683)
0925 15:16:32.477250 (+ 760us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:32.477257 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:32.478488 (+ 1231us) server_negotiation.cc:407] Connection header received
0925 15:16:32.478685 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:32.478690 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:32.478836 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:32.479036 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:16:32.479580 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:32.481112 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:32.483222 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:32.484285 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:32.484411 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:32.484633 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:32.484811 (+ 178us) server_negotiation.cc:299] Negotiation successful
0925 15:16:32.484966 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":434,"threads_started":1}
I20260925 15:16:32.486058 28255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:32.463753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.1:43327 (local address 127.0.0.1:49590)
0925 15:16:32.475645 (+ 11892us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:32.475671 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:32.475768 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:32.477475 (+ 1707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:32.477479 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:32.477506 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:16:32.477997 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:32.478010 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:32.480151 (+ 2141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:32.480155 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:32.482128 (+ 1973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:32.482139 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:32.483996 (+ 1857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:32.484003 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:32.484302 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:32.485440 (+ 1138us) client_negotiation.cc:769] Sending connection context
0925 15:16:32.485742 (+ 302us) client_negotiation.cc:240] Negotiation successful
0925 15:16:32.485826 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11779,"mutex_wait_us":1154,"thread_start_us":12100,"threads_started":1}
I20260925 15:16:32.485349 28258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:32.476437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.3:46683 (local address 127.0.0.1:36362)
0925 15:16:32.478376 (+ 1939us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:32.478389 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:32.478524 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:32.479012 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:32.479017 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:32.479040 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:16:32.479412 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:32.479420 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:32.481290 (+ 1870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:32.481295 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:32.483051 (+ 1756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:32.483062 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:32.484458 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:32.484464 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:32.484622 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:32.484667 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 15:16:32.484771 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 15:16:32.484903 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1833,"mutex_wait_us":1,"thread_start_us":1636,"threads_started":1}
I20260925 15:16:32.486315 28257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:32.464253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49590 (local address 127.12.8.1:43327)
0925 15:16:32.476876 (+ 12623us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:32.476886 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:32.476915 (+ 29us) server_negotiation.cc:407] Connection header received
0925 15:16:32.477017 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:32.477025 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:32.477271 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:32.477488 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:16:32.478253 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:32.479740 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:32.482308 (+ 2568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:32.483797 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:32.483951 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:32.484208 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:32.485817 (+ 1609us) server_negotiation.cc:299] Negotiation successful
0925 15:16:32.485971 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12445,"thread_start_us":12275,"threads_started":1}
I20260925 15:16:32.489084 28044 tablet_service.cc:1467] Processing CreateTablet for tablet 545bc6cc14a147a2a0d0dbdae92f66ab (DEFAULT_TABLE table=test-table [id=b91198ef373748d59ba3ac6c288155cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:16:32.490335 28044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 545bc6cc14a147a2a0d0dbdae92f66ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:16:32.489866 28177 tablet_service.cc:1467] Processing CreateTablet for tablet 545bc6cc14a147a2a0d0dbdae92f66ab (DEFAULT_TABLE table=test-table [id=b91198ef373748d59ba3ac6c288155cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:16:32.489866 27910 tablet_service.cc:1467] Processing CreateTablet for tablet 545bc6cc14a147a2a0d0dbdae92f66ab (DEFAULT_TABLE table=test-table [id=b91198ef373748d59ba3ac6c288155cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:16:32.491218 27910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 545bc6cc14a147a2a0d0dbdae92f66ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:16:32.491242 28177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 545bc6cc14a147a2a0d0dbdae92f66ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:16:32.501142 28260 tablet_bootstrap.cc:492] T 545bc6cc14a147a2a0d0dbdae92f66ab P 7bba3fc3f1e84d7c8254a77f05fd82df: Bootstrap starting.
I20260925 15:16:32.504660 28260 tablet_bootstrap.cc:654] T 545bc6cc14a147a2a0d0dbdae92f66ab P 7bba3fc3f1e84d7c8254a77f05fd82df: Neither blocks nor log segments found. Creating new log.
I20260925 15:16:32.505501 28261 tablet_bootstrap.cc:492] T 545bc6cc14a147a2a0d0dbdae92f66ab P a2cf2ac0a7a14b00986fab5569735fb8: Bootstrap starting.
I20260925 15:16:32.506444 28260 log.cc:824] T 545bc6cc14a147a2a0d0dbdae92f66ab P 7bba3fc3f1e84d7c8254a77f05fd82df: Log is configured to *not* fsync() on all Append() calls
I20260925 15:16:32.507020 28262 tablet_bootstrap.cc:492] T 545bc6cc14a147a2a0d0dbdae92f66ab P 232b8a44ea8e4bd39da1eed7fcb322e4: Bootstrap starting.
I20260925 15:16:32.509433 28261 tablet_bootstrap.cc:654] T 545bc6cc14a147a2a0d0dbdae92f66ab P a2cf2ac0a7a14b00986fab5569735fb8: Neither blocks nor log segments found. Creating new log.
I20260925 15:16:32.509910 28262 tablet_bootstrap.cc:654] T 545bc6cc14a147a2a0d0dbdae92f66ab P 232b8a44ea8e4bd39da1eed7fcb322e4: Neither blocks nor log segments found. Creating new log.
I20260925 15:16:32.510749 28261 log.cc:824] T 545bc6cc14a147a2a0d0dbdae92f66ab P a2cf2ac0a7a14b00986fab5569735fb8: Log is configured to *not* fsync() on all Append() calls
I20260925 15:16:32.511093 28262 log.cc:824] T 545bc6cc14a147a2a0d0dbdae92f66ab P 232b8a44ea8e4bd39da1eed7fcb322e4: Log is configured to *not* fsync() on all Append() calls
I20260925 15:16:32.512691 28260 tablet_bootstrap.cc:492] T 545bc6cc14a147a2a0d0dbdae92f66ab P 7bba3fc3f1e84d7c8254a77f05fd82df: No bootstrap required, opened a new log
I20260925 15:16:32.512982 28260 ts_tablet_manager.cc:1397] T 545bc6cc14a147a2a0d0dbdae92f66ab P 7bba3fc3f1e84d7c8254a77f05fd82df: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260925 15:16:32.513687 28261 tablet_bootstrap.cc:492] T 545bc6cc14a147a2a0d0dbdae92f66ab P a2cf2ac0a7a14b00986fab5569735fb8: No bootstrap required, opened a new log
I20260925 15:16:32.513939 28261 ts_tablet_manager.cc:1397] T 545bc6cc14a147a2a0d0dbdae92f66ab P a2cf2ac0a7a14b00986fab5569735fb8: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260925 15:16:32.514048 28262 tablet_bootstrap.cc:492] T 545bc6cc14a147a2a0d0dbdae92f66ab P 232b8a44ea8e4bd39da1eed7fcb322e4: No bootstrap required, opened a new log
I20260925 15:16:32.514310 28262 ts_tablet_manager.cc:1397] T 545bc6cc14a147a2a0d0dbdae92f66ab P 232b8a44ea8e4bd39da1eed7fcb322e4: Time spent bootstrapping tablet: real 0.007s user 0.007s sys 0.000s
I20260925 15:16:32.520608 28260 raft_consensus.cc:348] T 545bc6cc14a147a2a0d0dbdae92f66ab P 7bba3fc3f1e84d7c8254a77f05fd82df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bba3fc3f1e84d7c8254a77f05fd82df" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 45177 } } peers { permanent_uuid: "232b8a44ea8e4bd39da1eed7fcb322e4" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 43327 } } peers { permanent_uuid: "a2cf2ac0a7a14b00986fab5569735fb8" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 46683 } }
I20260925 15:16:32.521106 28260 raft_consensus.cc:374] T 545bc6cc14a147a2a0d0dbdae92f66ab P 7bba3fc3f1e84d7c8254a77f05fd82df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:16:32.521238 28260 raft_consensus.cc:729] T 545bc6cc14a147a2a0d0dbdae92f66ab P 7bba3fc3f1e84d7c8254a77f05fd82df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bba3fc3f1e84d7c8254a77f05fd82df, State: Initialized, Role: FOLLOWER
I20260925 15:16:32.521999 28260 consensus_queue.cc:260] T 545bc6cc14a147a2a0d0dbdae92f66ab P 7bba3fc3f1e84d7c8254a77f05fd82df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bba3fc3f1e84d7c8254a77f05fd82df" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 45177 } } peers { permanent_uuid: "232b8a44ea8e4bd39da1eed7fcb322e4" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 43327 } } peers { permanent_uuid: "a2cf2ac0a7a14b00986fab5569735fb8" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 46683 } }
I20260925 15:16:32.523290 28109 heartbeater.cc:503] Master 127.12.8.62:36947 was elected leader, sending a full tablet report...
I20260925 15:16:32.524294 28260 ts_tablet_manager.cc:1428] T 545bc6cc14a147a2a0d0dbdae92f66ab P 7bba3fc3f1e84d7c8254a77f05fd82df: Time spent starting tablet: real 0.011s user 0.003s sys 0.009s
I20260925 15:16:32.524946 28261 raft_consensus.cc:348] T 545bc6cc14a147a2a0d0dbdae92f66ab P a2cf2ac0a7a14b00986fab5569735fb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bba3fc3f1e84d7c8254a77f05fd82df" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 45177 } } peers { permanent_uuid: "232b8a44ea8e4bd39da1eed7fcb322e4" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 43327 } } peers { permanent_uuid: "a2cf2ac0a7a14b00986fab5569735fb8" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 46683 } }
I20260925 15:16:32.524998 28262 raft_consensus.cc:348] T 545bc6cc14a147a2a0d0dbdae92f66ab P 232b8a44ea8e4bd39da1eed7fcb322e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bba3fc3f1e84d7c8254a77f05fd82df" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 45177 } } peers { permanent_uuid: "232b8a44ea8e4bd39da1eed7fcb322e4" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 43327 } } peers { permanent_uuid: "a2cf2ac0a7a14b00986fab5569735fb8" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 46683 } }
I20260925 15:16:32.525394 28262 raft_consensus.cc:374] T 545bc6cc14a147a2a0d0dbdae92f66ab P 232b8a44ea8e4bd39da1eed7fcb322e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:16:32.525394 28261 raft_consensus.cc:374] T 545bc6cc14a147a2a0d0dbdae92f66ab P a2cf2ac0a7a14b00986fab5569735fb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:16:32.525501 28262 raft_consensus.cc:729] T 545bc6cc14a147a2a0d0dbdae92f66ab P 232b8a44ea8e4bd39da1eed7fcb322e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 232b8a44ea8e4bd39da1eed7fcb322e4, State: Initialized, Role: FOLLOWER
I20260925 15:16:32.525497 28261 raft_consensus.cc:729] T 545bc6cc14a147a2a0d0dbdae92f66ab P a2cf2ac0a7a14b00986fab5569735fb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2cf2ac0a7a14b00986fab5569735fb8, State: Initialized, Role: FOLLOWER
I20260925 15:16:32.526088 28262 consensus_queue.cc:260] T 545bc6cc14a147a2a0d0dbdae92f66ab P 232b8a44ea8e4bd39da1eed7fcb322e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bba3fc3f1e84d7c8254a77f05fd82df" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 45177 } } peers { permanent_uuid: "232b8a44ea8e4bd39da1eed7fcb322e4" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 43327 } } peers { permanent_uuid: "a2cf2ac0a7a14b00986fab5569735fb8" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 46683 } }
I20260925 15:16:32.526089 28261 consensus_queue.cc:260] T 545bc6cc14a147a2a0d0dbdae92f66ab P a2cf2ac0a7a14b00986fab5569735fb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bba3fc3f1e84d7c8254a77f05fd82df" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 45177 } } peers { permanent_uuid: "232b8a44ea8e4bd39da1eed7fcb322e4" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 43327 } } peers { permanent_uuid: "a2cf2ac0a7a14b00986fab5569735fb8" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 46683 } }
I20260925 15:16:32.527719 28242 heartbeater.cc:503] Master 127.12.8.62:36947 was elected leader, sending a full tablet report...
I20260925 15:16:32.527915 28261 ts_tablet_manager.cc:1428] T 545bc6cc14a147a2a0d0dbdae92f66ab P a2cf2ac0a7a14b00986fab5569735fb8: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260925 15:16:32.528213 28262 ts_tablet_manager.cc:1428] T 545bc6cc14a147a2a0d0dbdae92f66ab P 232b8a44ea8e4bd39da1eed7fcb322e4: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
W20260925 15:16:32.629094 28110 tablet.cc:2369] T 545bc6cc14a147a2a0d0dbdae92f66ab P 7bba3fc3f1e84d7c8254a77f05fd82df: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:16:32.632612 27976 tablet.cc:2369] T 545bc6cc14a147a2a0d0dbdae92f66ab P 232b8a44ea8e4bd39da1eed7fcb322e4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:16:32.642765 28243 tablet.cc:2369] T 545bc6cc14a147a2a0d0dbdae92f66ab P a2cf2ac0a7a14b00986fab5569735fb8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:16:32.652810 28268 raft_consensus.cc:482] T 545bc6cc14a147a2a0d0dbdae92f66ab P 232b8a44ea8e4bd39da1eed7fcb322e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:16:32.653069 28268 raft_consensus.cc:504] T 545bc6cc14a147a2a0d0dbdae92f66ab P 232b8a44ea8e4bd39da1eed7fcb322e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bba3fc3f1e84d7c8254a77f05fd82df" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 45177 } } peers { permanent_uuid: "232b8a44ea8e4bd39da1eed7fcb322e4" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 43327 } } peers { permanent_uuid: "a2cf2ac0a7a14b00986fab5569735fb8" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 46683 } }
I20260925 15:16:32.654830 28268 leader_election.cc:288] T 545bc6cc14a147a2a0d0dbdae92f66ab P 232b8a44ea8e4bd39da1eed7fcb322e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7bba3fc3f1e84d7c8254a77f05fd82df (127.12.8.2:45177), a2cf2ac0a7a14b00986fab5569735fb8 (127.12.8.3:46683)
I20260925 15:16:32.661623 28256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:32.655302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.1:36451 (local address 127.12.8.2:45177)
0925 15:16:32.655533 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:32.655542 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:32.656204 (+ 662us) server_negotiation.cc:407] Connection header received
0925 15:16:32.656448 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:32.656452 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:32.656511 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:32.656609 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:16:32.657176 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:32.657861 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:32.659810 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:32.660653 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:32.660793 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:32.660904 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:32.661245 (+ 341us) server_negotiation.cc:299] Negotiation successful
0925 15:16:32.661370 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 15:16:32.661765 28269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:32.655003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.2:45177 (local address 127.12.8.1:36451)
0925 15:16:32.656071 (+ 1068us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:32.656088 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:32.656289 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:32.656632 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:32.656636 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:32.656662 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:16:32.657019 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:32.657030 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:32.657988 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:32.657995 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:32.659652 (+ 1657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:32.659664 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:32.660943 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:32.660951 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:32.661077 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:32.661100 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 15:16:32.661220 (+ 120us) client_negotiation.cc:240] Negotiation successful
0925 15:16:32.661348 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":945,"mutex_wait_us":50,"thread_start_us":699,"threads_started":1}
I20260925 15:16:32.661831 28259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:32.655421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.1:46345 (local address 127.12.8.3:46683)
0925 15:16:32.655729 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:32.655737 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:32.656305 (+ 568us) server_negotiation.cc:407] Connection header received
0925 15:16:32.656509 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:32.656513 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:32.656569 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:32.656678 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:16:32.657300 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:32.658017 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:32.659835 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:32.660592 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:32.660748 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:32.660856 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:32.661433 (+ 577us) server_negotiation.cc:299] Negotiation successful
0925 15:16:32.661571 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136}
I20260925 15:16:32.662676 28270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:32.655281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.3:46683 (local address 127.12.8.1:46345)
0925 15:16:32.656189 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:32.656205 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:32.656369 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:32.656690 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:32.656694 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:32.656720 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:16:32.657156 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:32.657164 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:32.658171 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:32.658176 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:32.659677 (+ 1501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:32.659688 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:32.661036 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:32.661047 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:32.661242 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:32.661279 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 15:16:32.662108 (+ 829us) client_negotiation.cc:240] Negotiation successful
0925 15:16:32.662264 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":787,"mutex_wait_us":5,"thread_start_us":434,"threads_started":1}
I20260925 15:16:32.663308 28064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "545bc6cc14a147a2a0d0dbdae92f66ab" candidate_uuid: "232b8a44ea8e4bd39da1eed7fcb322e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bba3fc3f1e84d7c8254a77f05fd82df" is_pre_election: true
I20260925 15:16:32.663710 28197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "545bc6cc14a147a2a0d0dbdae92f66ab" candidate_uuid: "232b8a44ea8e4bd39da1eed7fcb322e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2cf2ac0a7a14b00986fab5569735fb8" is_pre_election: true
I20260925 15:16:32.663935 28064 raft_consensus.cc:2454] T 545bc6cc14a147a2a0d0dbdae92f66ab P 7bba3fc3f1e84d7c8254a77f05fd82df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 232b8a44ea8e4bd39da1eed7fcb322e4 in term 0.
I20260925 15:16:32.664249 28197 raft_consensus.cc:2454] T 545bc6cc14a147a2a0d0dbdae92f66ab P a2cf2ac0a7a14b00986fab5569735fb8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 232b8a44ea8e4bd39da1eed7fcb322e4 in term 0.
I20260925 15:16:32.664732 27866 leader_election.cc:302] T 545bc6cc14a147a2a0d0dbdae92f66ab P 232b8a44ea8e4bd39da1eed7fcb322e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 232b8a44ea8e4bd39da1eed7fcb322e4, 7bba3fc3f1e84d7c8254a77f05fd82df; no voters:
I20260925 15:16:32.665149 28268 raft_consensus.cc:2781] T 545bc6cc14a147a2a0d0dbdae92f66ab P 232b8a44ea8e4bd39da1eed7fcb322e4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:16:32.665313 28268 raft_consensus.cc:482] T 545bc6cc14a147a2a0d0dbdae92f66ab P 232b8a44ea8e4bd39da1eed7fcb322e4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:16:32.665429 28268 raft_consensus.cc:3037] T 545bc6cc14a147a2a0d0dbdae92f66ab P 232b8a44ea8e4bd39da1eed7fcb322e4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:16:32.667655 28268 raft_consensus.cc:504] T 545bc6cc14a147a2a0d0dbdae92f66ab P 232b8a44ea8e4bd39da1eed7fcb322e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bba3fc3f1e84d7c8254a77f05fd82df" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 45177 } } peers { permanent_uuid: "232b8a44ea8e4bd39da1eed7fcb322e4" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 43327 } } peers { permanent_uuid: "a2cf2ac0a7a14b00986fab5569735fb8" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 46683 } }
I20260925 15:16:32.668287 28268 leader_election.cc:288] T 545bc6cc14a147a2a0d0dbdae92f66ab P 232b8a44ea8e4bd39da1eed7fcb322e4 [CANDIDATE]: Term 1 election: Requested vote from peers 7bba3fc3f1e84d7c8254a77f05fd82df (127.12.8.2:45177), a2cf2ac0a7a14b00986fab5569735fb8 (127.12.8.3:46683)
I20260925 15:16:32.668666 28064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "545bc6cc14a147a2a0d0dbdae92f66ab" candidate_uuid: "232b8a44ea8e4bd39da1eed7fcb322e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bba3fc3f1e84d7c8254a77f05fd82df"
I20260925 15:16:32.668797 28197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "545bc6cc14a147a2a0d0dbdae92f66ab" candidate_uuid: "232b8a44ea8e4bd39da1eed7fcb322e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2cf2ac0a7a14b00986fab5569735fb8"
I20260925 15:16:32.668892 28064 raft_consensus.cc:3037] T 545bc6cc14a147a2a0d0dbdae92f66ab P 7bba3fc3f1e84d7c8254a77f05fd82df [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:16:32.668995 28197 raft_consensus.cc:3037] T 545bc6cc14a147a2a0d0dbdae92f66ab P a2cf2ac0a7a14b00986fab5569735fb8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:16:32.672179 28197 raft_consensus.cc:2454] T 545bc6cc14a147a2a0d0dbdae92f66ab P a2cf2ac0a7a14b00986fab5569735fb8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 232b8a44ea8e4bd39da1eed7fcb322e4 in term 1.
I20260925 15:16:32.672179 28064 raft_consensus.cc:2454] T 545bc6cc14a147a2a0d0dbdae92f66ab P 7bba3fc3f1e84d7c8254a77f05fd82df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 232b8a44ea8e4bd39da1eed7fcb322e4 in term 1.
I20260925 15:16:32.672590 27863 leader_election.cc:302] T 545bc6cc14a147a2a0d0dbdae92f66ab P 232b8a44ea8e4bd39da1eed7fcb322e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 232b8a44ea8e4bd39da1eed7fcb322e4, a2cf2ac0a7a14b00986fab5569735fb8; no voters:
I20260925 15:16:32.672932 28268 raft_consensus.cc:2781] T 545bc6cc14a147a2a0d0dbdae92f66ab P 232b8a44ea8e4bd39da1eed7fcb322e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:16:32.673415 28268 raft_consensus.cc:686] T 545bc6cc14a147a2a0d0dbdae92f66ab P 232b8a44ea8e4bd39da1eed7fcb322e4 [term 1 LEADER]: Becoming Leader. State: Replica: 232b8a44ea8e4bd39da1eed7fcb322e4, State: Running, Role: LEADER
I20260925 15:16:32.674124 28268 consensus_queue.cc:237] T 545bc6cc14a147a2a0d0dbdae92f66ab P 232b8a44ea8e4bd39da1eed7fcb322e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bba3fc3f1e84d7c8254a77f05fd82df" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 45177 } } peers { permanent_uuid: "232b8a44ea8e4bd39da1eed7fcb322e4" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 43327 } } peers { permanent_uuid: "a2cf2ac0a7a14b00986fab5569735fb8" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 46683 } }
I20260925 15:16:32.678818 27785 catalog_manager.cc:5495] T 545bc6cc14a147a2a0d0dbdae92f66ab P 232b8a44ea8e4bd39da1eed7fcb322e4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 232b8a44ea8e4bd39da1eed7fcb322e4 (127.12.8.1). New cstate: current_term: 1 leader_uuid: "232b8a44ea8e4bd39da1eed7fcb322e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bba3fc3f1e84d7c8254a77f05fd82df" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 45177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "232b8a44ea8e4bd39da1eed7fcb322e4" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 43327 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2cf2ac0a7a14b00986fab5569735fb8" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 46683 } health_report { overall_health: UNKNOWN } } }
I20260925 15:16:32.720849 28257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:32.715364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49600 (local address 127.12.8.1:43327)
0925 15:16:32.715600 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:32.715605 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:32.715623 (+ 18us) server_negotiation.cc:407] Connection header received
0925 15:16:32.715819 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:32.715822 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:32.715898 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:32.716031 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:16:32.716379 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:32.717202 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:32.718821 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:32.719216 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:32.719309 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:32.719773 (+ 464us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:16:32.720263 (+ 490us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:16:32.720371 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:32.720467 (+ 96us) server_negotiation.cc:299] Negotiation successful
0925 15:16:32.720530 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260925 15:16:32.729359 28197 raft_consensus.cc:1261] T 545bc6cc14a147a2a0d0dbdae92f66ab P a2cf2ac0a7a14b00986fab5569735fb8 [term 1 FOLLOWER]: Refusing update from remote peer 232b8a44ea8e4bd39da1eed7fcb322e4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:16:32.729411 28064 raft_consensus.cc:1261] T 545bc6cc14a147a2a0d0dbdae92f66ab P 7bba3fc3f1e84d7c8254a77f05fd82df [term 1 FOLLOWER]: Refusing update from remote peer 232b8a44ea8e4bd39da1eed7fcb322e4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:16:32.730321 28271 consensus_queue.cc:1035] T 545bc6cc14a147a2a0d0dbdae92f66ab P 232b8a44ea8e4bd39da1eed7fcb322e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2cf2ac0a7a14b00986fab5569735fb8" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 46683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:16:32.730742 28268 consensus_queue.cc:1035] T 545bc6cc14a147a2a0d0dbdae92f66ab P 232b8a44ea8e4bd39da1eed7fcb322e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bba3fc3f1e84d7c8254a77f05fd82df" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 45177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:16:32.754849 27784 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34808:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:16:32.755177 27784 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34808:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:16:32.758939 27784 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 46041bb901664061b44eb699831c4e55: Sending DeleteTablet for 3 replicas of tablet 545bc6cc14a147a2a0d0dbdae92f66ab
I20260925 15:16:32.760059 28044 tablet_service.cc:1514] Processing DeleteTablet for tablet 545bc6cc14a147a2a0d0dbdae92f66ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:16:32 UTC) from {username='kudu', principal='kudu/127.12.8.62@KRBTEST.COM'} at 127.0.0.1:37070
I20260925 15:16:32.760180 27910 tablet_service.cc:1514] Processing DeleteTablet for tablet 545bc6cc14a147a2a0d0dbdae92f66ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:16:32 UTC) from {username='kudu', principal='kudu/127.12.8.62@KRBTEST.COM'} at 127.0.0.1:49590
I20260925 15:16:32.760175 28177 tablet_service.cc:1514] Processing DeleteTablet for tablet 545bc6cc14a147a2a0d0dbdae92f66ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:16:32 UTC) from {username='kudu', principal='kudu/127.12.8.62@KRBTEST.COM'} at 127.0.0.1:36362
I20260925 15:16:32.761274 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 27847
I20260925 15:16:32.772504 28285 tablet_replica.cc:330] T 545bc6cc14a147a2a0d0dbdae92f66ab P a2cf2ac0a7a14b00986fab5569735fb8: stopping tablet replica
I20260925 15:16:32.773180 28284 tablet_replica.cc:330] T 545bc6cc14a147a2a0d0dbdae92f66ab P 7bba3fc3f1e84d7c8254a77f05fd82df: stopping tablet replica
I20260925 15:16:32.773759 28284 raft_consensus.cc:2229] T 545bc6cc14a147a2a0d0dbdae92f66ab P 7bba3fc3f1e84d7c8254a77f05fd82df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:16:32.774379 28284 raft_consensus.cc:2258] T 545bc6cc14a147a2a0d0dbdae92f66ab P 7bba3fc3f1e84d7c8254a77f05fd82df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:16:32.775244 28286 tablet_replica.cc:330] T 545bc6cc14a147a2a0d0dbdae92f66ab P 232b8a44ea8e4bd39da1eed7fcb322e4: stopping tablet replica
I20260925 15:16:32.775744 28286 raft_consensus.cc:2229] T 545bc6cc14a147a2a0d0dbdae92f66ab P 232b8a44ea8e4bd39da1eed7fcb322e4 [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:16:32.776528 28286 raft_consensus.cc:2258] T 545bc6cc14a147a2a0d0dbdae92f66ab P 232b8a44ea8e4bd39da1eed7fcb322e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:16:32.777423 28285 raft_consensus.cc:2229] T 545bc6cc14a147a2a0d0dbdae92f66ab P a2cf2ac0a7a14b00986fab5569735fb8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:16:32.777999 28285 raft_consensus.cc:2258] T 545bc6cc14a147a2a0d0dbdae92f66ab P a2cf2ac0a7a14b00986fab5569735fb8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:16:32.780257 28284 ts_tablet_manager.cc:1905] T 545bc6cc14a147a2a0d0dbdae92f66ab P 7bba3fc3f1e84d7c8254a77f05fd82df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:16:32.780254 28285 ts_tablet_manager.cc:1905] T 545bc6cc14a147a2a0d0dbdae92f66ab P a2cf2ac0a7a14b00986fab5569735fb8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:16:32.780445 28286 ts_tablet_manager.cc:1905] T 545bc6cc14a147a2a0d0dbdae92f66ab P 232b8a44ea8e4bd39da1eed7fcb322e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:16:32.787164 28284 ts_tablet_manager.cc:1918] T 545bc6cc14a147a2a0d0dbdae92f66ab P 7bba3fc3f1e84d7c8254a77f05fd82df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:16:32.787331 28285 ts_tablet_manager.cc:1918] T 545bc6cc14a147a2a0d0dbdae92f66ab P a2cf2ac0a7a14b00986fab5569735fb8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:16:32.787390 28284 log.cc:1196] T 545bc6cc14a147a2a0d0dbdae92f66ab P 7bba3fc3f1e84d7c8254a77f05fd82df: Deleting WAL directory at /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-1/wal/wals/545bc6cc14a147a2a0d0dbdae92f66ab
I20260925 15:16:32.787514 28285 log.cc:1196] T 545bc6cc14a147a2a0d0dbdae92f66ab P a2cf2ac0a7a14b00986fab5569735fb8: Deleting WAL directory at /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-2/wal/wals/545bc6cc14a147a2a0d0dbdae92f66ab
I20260925 15:16:32.788045 28285 ts_tablet_manager.cc:1939] T 545bc6cc14a147a2a0d0dbdae92f66ab P a2cf2ac0a7a14b00986fab5569735fb8: Deleting consensus metadata
I20260925 15:16:32.788074 28284 ts_tablet_manager.cc:1939] T 545bc6cc14a147a2a0d0dbdae92f66ab P 7bba3fc3f1e84d7c8254a77f05fd82df: Deleting consensus metadata
I20260925 15:16:32.789956 27770 catalog_manager.cc:4841] TS a2cf2ac0a7a14b00986fab5569735fb8 (127.12.8.3:46683): tablet 545bc6cc14a147a2a0d0dbdae92f66ab (table test-table [id=b91198ef373748d59ba3ac6c288155cb]) successfully deleted
I20260925 15:16:32.789921 27773 catalog_manager.cc:4841] TS 7bba3fc3f1e84d7c8254a77f05fd82df (127.12.8.2:45177): tablet 545bc6cc14a147a2a0d0dbdae92f66ab (table test-table [id=b91198ef373748d59ba3ac6c288155cb]) successfully deleted
I20260925 15:16:32.791913 28286 ts_tablet_manager.cc:1918] T 545bc6cc14a147a2a0d0dbdae92f66ab P 232b8a44ea8e4bd39da1eed7fcb322e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:16:32.792107 28286 log.cc:1196] T 545bc6cc14a147a2a0d0dbdae92f66ab P 232b8a44ea8e4bd39da1eed7fcb322e4: Deleting WAL directory at /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-0/wal/wals/545bc6cc14a147a2a0d0dbdae92f66ab
I20260925 15:16:32.792725 28286 ts_tablet_manager.cc:1939] T 545bc6cc14a147a2a0d0dbdae92f66ab P 232b8a44ea8e4bd39da1eed7fcb322e4: Deleting consensus metadata
I20260925 15:16:32.794551 27770 catalog_manager.cc:4841] TS 232b8a44ea8e4bd39da1eed7fcb322e4 (127.12.8.1:43327): tablet 545bc6cc14a147a2a0d0dbdae92f66ab (table test-table [id=b91198ef373748d59ba3ac6c288155cb]) successfully deleted
Sep 25 15:16:32 dist-test-slave-p468 krb5kdc[27741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.1@KRBTEST.COM
I20260925 15:16:32.803790 28257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:32.762689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49602 (local address 127.12.8.1:43327)
0925 15:16:32.762914 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:32.762923 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:32.778640 (+ 15717us) server_negotiation.cc:407] Connection header received
0925 15:16:32.778887 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:32.778895 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:32.778986 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:32.779091 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:32.788315 (+ 9224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:32.789416 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:32.791140 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:32.791858 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:32.791995 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:32.797673 (+ 5678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:16:32.797683 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:16:32.797700 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:16:32.797736 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:16:32.800605 (+ 2869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:32.801452 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:32.801459 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:32.801461 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:32.801613 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:32.802070 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:32.802074 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:32.802076 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:32.802539 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:16:32.802776 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:32.803262 (+ 486us) server_negotiation.cc:299] Negotiation successful
0925 15:16:32.803439 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 15:16:32.804564 27970 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:16:32.904465 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 27847
I20260925 15:16:32.923120 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 27981
Sep 25 15:16:32 dist-test-slave-p468 krb5kdc[27741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.2@KRBTEST.COM
I20260925 15:16:32.940160 28256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:32.923936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37074 (local address 127.12.8.2:45177)
0925 15:16:32.924087 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:32.924093 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:32.924173 (+ 80us) server_negotiation.cc:407] Connection header received
0925 15:16:32.924398 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:32.924405 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:32.924511 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:32.924604 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:32.926243 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:32.927013 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:32.928467 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:32.928849 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:32.928947 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:32.933917 (+ 4970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:16:32.933933 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:16:32.933968 (+ 35us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:16:32.934020 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:16:32.937388 (+ 3368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:32.938043 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:32.938053 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:32.938057 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:32.938338 (+ 281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:32.938746 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:32.938750 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:32.938752 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:32.939112 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:16:32.939330 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:32.939612 (+ 282us) server_negotiation.cc:299] Negotiation successful
0925 15:16:32.939781 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 15:16:32.940851 28104 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:16:33.029702 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 27981
I20260925 15:16:33.046379 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 28114
Sep 25 15:16:33 dist-test-slave-p468 krb5kdc[27741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.3@KRBTEST.COM
I20260925 15:16:33.061682 28259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:33.047213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36376 (local address 127.12.8.3:46683)
0925 15:16:33.047404 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:33.047410 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:33.047430 (+ 20us) server_negotiation.cc:407] Connection header received
0925 15:16:33.047626 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:33.047632 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:33.047718 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:33.047827 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:33.049064 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:33.049778 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:33.051198 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:33.051584 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:33.051713 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:33.056115 (+ 4402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:16:33.056129 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:16:33.056145 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:16:33.056176 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:16:33.059010 (+ 2834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:33.059683 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:33.059698 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:33.059701 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:33.059838 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:33.060209 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:33.060217 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:33.060219 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:33.060540 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:16:33.060834 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:33.061142 (+ 308us) server_negotiation.cc:299] Negotiation successful
0925 15:16:33.061357 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 15:16:33.062559 28237 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:16:33.150705 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 28114
I20260925 15:16:33.167423 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 27754
I20260925 15:16:33.168711 27816 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:16:33.281134 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 27754
I20260925 15:16:53.302111 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.62:36947
--webserver_interface=127.12.8.62
--webserver_port=41499
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab
--principal=kudu/127.12.8.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:38153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.62:36947
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:16:53.622382 28292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:16:53.622720 28292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:16:53.622871 28292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:16:53.629964 28292 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 15:16:53.630095 28292 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 15:16:53.630184 28292 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 15:16:53.630287 28292 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:16:53.630386 28292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:16:53.630498 28292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:16:53.630582 28292 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:16:53.630671 28292 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:16:53.640256 28292 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:38153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.12.8.62:36947
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab
--principal=kudu/127.12.8.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.62:36947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.8.62
--webserver_port=41499
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:16:53.641561 28292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:16:53.642910 28292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:16:53.665230 28301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:16:53.665171 28299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:16:53.665279 28298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:16:53.665982 28292 server_base.cc:1034] running on GCE node
I20260925 15:16:53.666926 28292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:16:53.668496 28292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:16:53.669720 28292 hybrid_clock.cc:648] HybridClock initialized: now 1790349413669644 us; error 53 us; skew 500 ppm
I20260925 15:16:53.670141 28292 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:16:53 dist-test-slave-p468 krb5kdc[27741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349413, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:16:53.678741 28292 init.cc:377] Logged in from keytab as kudu/127.12.8.62@KRBTEST.COM (short username kudu)
I20260925 15:16:53.680436 28292 webserver.cc:460] Webserver started at http://127.12.8.62:41499/ using document root <none> and password file <none>
I20260925 15:16:53.681217 28292 fs_manager.cc:359] Metadata directory not provided
I20260925 15:16:53.681358 28292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:16:53.687582 28292 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 15:16:53.690568 28308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:53.691927 28292 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:16:53.692135 28292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/master-0/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/master-0/wal
uuid: "46041bb901664061b44eb699831c4e55"
format_stamp: "Formatted at 2026-09-25 15:16:30 on dist-test-slave-p468"
server_key: "d1172c8c8c128130cd60dc56711820cc"
server_key_iv: "0884fac03bac734537c56062624c1806"
server_key_version: "encryptionkey@0"
I20260925 15:16:53.693184 28292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:16:53.710956 28292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:16:53.715431 28292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:16:53.715721 28292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:16:53.735140 28292 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.62:36947
I20260925 15:16:53.735162 28359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.62:36947 every 8 connection(s)
I20260925 15:16:53.736691 28292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
I20260925 15:16:53.744154 28360 sys_catalog.cc:263] Verifying existing consensus state
I20260925 15:16:53.745174 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 28292
I20260925 15:16:53.746233 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.1:43327
--local_ip_for_outbound_sockets=127.12.8.1
--tserver_master_addrs=127.12.8.62:36947
--webserver_port=45247
--webserver_interface=127.12.8.1
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.1.keytab
--principal=kudu/127.12.8.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:38153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:16:53.749275 28360 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46041bb901664061b44eb699831c4e55: Bootstrap starting.
I20260925 15:16:53.771170 28360 log.cc:824] T 00000000000000000000000000000000 P 46041bb901664061b44eb699831c4e55: Log is configured to *not* fsync() on all Append() calls
I20260925 15:16:53.783599 28360 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46041bb901664061b44eb699831c4e55: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 15:16:53.784154 28360 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46041bb901664061b44eb699831c4e55: Bootstrap complete.
I20260925 15:16:53.793457 28360 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46041bb901664061b44eb699831c4e55 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46041bb901664061b44eb699831c4e55" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 36947 } }
I20260925 15:16:53.794353 28360 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46041bb901664061b44eb699831c4e55 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46041bb901664061b44eb699831c4e55, State: Initialized, Role: FOLLOWER
I20260925 15:16:53.794991 28360 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46041bb901664061b44eb699831c4e55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46041bb901664061b44eb699831c4e55" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 36947 } }
I20260925 15:16:53.795258 28360 raft_consensus.cc:388] T 00000000000000000000000000000000 P 46041bb901664061b44eb699831c4e55 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:16:53.795387 28360 raft_consensus.cc:482] T 00000000000000000000000000000000 P 46041bb901664061b44eb699831c4e55 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:16:53.795579 28360 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46041bb901664061b44eb699831c4e55 [term 1 FOLLOWER]: Advancing to term 2
I20260925 15:16:53.799839 28360 raft_consensus.cc:504] T 00000000000000000000000000000000 P 46041bb901664061b44eb699831c4e55 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46041bb901664061b44eb699831c4e55" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 36947 } }
I20260925 15:16:53.800407 28360 leader_election.cc:302] T 00000000000000000000000000000000 P 46041bb901664061b44eb699831c4e55 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 46041bb901664061b44eb699831c4e55; no voters:
I20260925 15:16:53.801064 28360 leader_election.cc:288] T 00000000000000000000000000000000 P 46041bb901664061b44eb699831c4e55 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 15:16:53.801215 28364 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 46041bb901664061b44eb699831c4e55 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 15:16:53.802511 28364 raft_consensus.cc:686] T 00000000000000000000000000000000 P 46041bb901664061b44eb699831c4e55 [term 2 LEADER]: Becoming Leader. State: Replica: 46041bb901664061b44eb699831c4e55, State: Running, Role: LEADER
I20260925 15:16:53.803192 28364 consensus_queue.cc:237] T 00000000000000000000000000000000 P 46041bb901664061b44eb699831c4e55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46041bb901664061b44eb699831c4e55" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 36947 } }
I20260925 15:16:53.803491 28360 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46041bb901664061b44eb699831c4e55 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:16:53.806043 28365 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46041bb901664061b44eb699831c4e55 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "46041bb901664061b44eb699831c4e55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46041bb901664061b44eb699831c4e55" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 36947 } } }
I20260925 15:16:53.806511 28365 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46041bb901664061b44eb699831c4e55 [sys.catalog]: This master's current role is: LEADER
I20260925 15:16:53.811246 28366 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46041bb901664061b44eb699831c4e55 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46041bb901664061b44eb699831c4e55. Latest consensus state: current_term: 2 leader_uuid: "46041bb901664061b44eb699831c4e55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46041bb901664061b44eb699831c4e55" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 36947 } } }
I20260925 15:16:53.811630 28366 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46041bb901664061b44eb699831c4e55 [sys.catalog]: This master's current role is: LEADER
I20260925 15:16:53.814604 28370 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:16:53.823355 28370 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:16:53.825060 28370 catalog_manager.cc:1259] Loaded cluster ID: 1a3299685a8e4f32a0950e03622ecab5
I20260925 15:16:53.825201 28370 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:16:53.829362 28370 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:16:53.830945 28370 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 46041bb901664061b44eb699831c4e55: Loaded TSK: 0
I20260925 15:16:53.867290 28370 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 46041bb901664061b44eb699831c4e55: Generated new TSK 1
I20260925 15:16:53.869725 28370 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 15:16:54.096544 28362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:16:54.096855 28362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:16:54.097030 28362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:16:54.104341 28362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:16:54.104478 28362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:16:54.104643 28362 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.1
I20260925 15:16:54.114351 28362 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:38153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.1.keytab
--principal=kudu/127.12.8.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.1:43327
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.8.1
--webserver_port=45247
--tserver_master_addrs=127.12.8.62:36947
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.1
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:16:54.115576 28362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:16:54.116885 28362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:16:54.131196 28387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:16:54.131569 28388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:16:54.132349 28390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:16:54.132457 28362 server_base.cc:1034] running on GCE node
I20260925 15:16:54.133054 28362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:16:54.134109 28362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:16:54.135346 28362 hybrid_clock.cc:648] HybridClock initialized: now 1790349414135305 us; error 38 us; skew 500 ppm
I20260925 15:16:54.135653 28362 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:16:54 dist-test-slave-p468 krb5kdc[27741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349414, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:16:54.138850 28362 init.cc:377] Logged in from keytab as kudu/127.12.8.1@KRBTEST.COM (short username kudu)
I20260925 15:16:54.140400 28362 webserver.cc:460] Webserver started at http://127.12.8.1:45247/ using document root <none> and password file <none>
I20260925 15:16:54.141054 28362 fs_manager.cc:359] Metadata directory not provided
I20260925 15:16:54.141204 28362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:16:54.146656 28362 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 15:16:54.149335 28397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:54.150750 28362 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:16:54.150918 28362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-0/wal
uuid: "232b8a44ea8e4bd39da1eed7fcb322e4"
format_stamp: "Formatted at 2026-09-25 15:16:31 on dist-test-slave-p468"
server_key: "3c2680983affd0ece3432972c9ed0d94"
server_key_iv: "7968c95a7ada86ad22f2d74947990827"
server_key_version: "encryptionkey@0"
I20260925 15:16:54.151866 28362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:16:54.163240 28362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:16:54.167296 28362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:16:54.167567 28362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:16:54.168572 28362 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:16:54.169874 28362 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:16:54.170004 28362 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:54.170118 28362 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:16:54.170218 28362 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:54.208441 28362 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.1:43327
I20260925 15:16:54.208513 28509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.1:43327 every 8 connection(s)
I20260925 15:16:54.210026 28362 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-0/data/info.pb
I20260925 15:16:54.215657 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 28362
I20260925 15:16:54.216621 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.2:45177
--local_ip_for_outbound_sockets=127.12.8.2
--tserver_master_addrs=127.12.8.62:36947
--webserver_port=36279
--webserver_interface=127.12.8.2
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.2.keytab
--principal=kudu/127.12.8.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:38153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 15:16:54 dist-test-slave-p468 krb5kdc[27741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349414, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.1@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
I20260925 15:16:54.239898 28513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:54.213761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:36947 (local address 127.12.8.1:42173)
0925 15:16:54.216508 (+ 2747us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:54.216567 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:54.219370 (+ 2803us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:54.221211 (+ 1841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:54.221221 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:54.221964 (+ 743us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:16:54.222793 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:54.222818 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:54.225321 (+ 2503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:54.225329 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:54.226778 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:54.226793 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:54.228078 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:54.228088 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:54.228441 (+ 353us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:54.229218 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:16:54.229292 (+ 74us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:16:54.231936 (+ 2644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:16:54.235568 (+ 3632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:16:54.235576 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:16:54.235591 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:16:54.236035 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:16:54.236456 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:16:54.236461 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:16:54.236465 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:16:54.236646 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:16:54.237448 (+ 802us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:16:54.237456 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:16:54.237784 (+ 328us) client_negotiation.cc:769] Sending connection context
0925 15:16:54.238043 (+ 259us) client_negotiation.cc:240] Negotiation successful
0925 15:16:54.238606 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2387,"thread_start_us":2145,"threads_started":1}
I20260925 15:16:54.239898 28512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:54.213477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.1:42173 (local address 127.12.8.62:36947)
0925 15:16:54.214851 (+ 1374us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:54.214869 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:54.216709 (+ 1840us) server_negotiation.cc:407] Connection header received
0925 15:16:54.220013 (+ 3304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:54.220036 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:54.220528 (+ 492us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:54.220965 (+ 437us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:54.223074 (+ 2109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:54.225128 (+ 2054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:54.226950 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:54.227771 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:54.227985 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:54.232265 (+ 4280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:16:54.232272 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:16:54.232287 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:16:54.232328 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:16:54.235421 (+ 3093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:54.236178 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:54.236184 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:54.236190 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:54.236344 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:54.236758 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:54.236768 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:54.236770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:54.237313 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:16:54.237514 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:54.238294 (+ 780us) server_negotiation.cc:299] Negotiation successful
0925 15:16:54.238701 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1125,"thread_start_us":977,"threads_started":1}
I20260925 15:16:54.242686 28510 heartbeater.cc:347] Connected to a master server at 127.12.8.62:36947
I20260925 15:16:54.243053 28510 heartbeater.cc:464] Registering TS with master...
I20260925 15:16:54.244000 28510 heartbeater.cc:511] Master 127.12.8.62:36947 requested a full tablet report, sending...
I20260925 15:16:54.246099 28325 ts_manager.cc:194] Registered new tserver with Master: 232b8a44ea8e4bd39da1eed7fcb322e4 (127.12.8.1:43327)
I20260925 15:16:54.247699 28325 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.1@KRBTEST.COM'} at 127.12.8.1:42173
W20260925 15:16:54.544399 28514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:16:54.544657 28514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:16:54.544798 28514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:16:54.552274 28514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:16:54.552395 28514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:16:54.552569 28514 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.2
I20260925 15:16:54.562003 28514 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:38153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.2.keytab
--principal=kudu/127.12.8.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.2:45177
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.8.2
--webserver_port=36279
--tserver_master_addrs=127.12.8.62:36947
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.2
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:16:54.563189 28514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:16:54.564455 28514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:16:54.577060 28519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:16:54.577004 28522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:16:54.577005 28520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:16:54.577482 28514 server_base.cc:1034] running on GCE node
I20260925 15:16:54.578795 28514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:16:54.579666 28514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:16:54.580876 28514 hybrid_clock.cc:648] HybridClock initialized: now 1790349414580836 us; error 49 us; skew 500 ppm
I20260925 15:16:54.581176 28514 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:16:54 dist-test-slave-p468 krb5kdc[27741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349414, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:16:54.584151 28514 init.cc:377] Logged in from keytab as kudu/127.12.8.2@KRBTEST.COM (short username kudu)
I20260925 15:16:54.585477 28514 webserver.cc:460] Webserver started at http://127.12.8.2:36279/ using document root <none> and password file <none>
I20260925 15:16:54.586050 28514 fs_manager.cc:359] Metadata directory not provided
I20260925 15:16:54.586159 28514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:16:54.591173 28514 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260925 15:16:54.593590 28529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:54.594897 28514 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 15:16:54.595022 28514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-1/wal
uuid: "7bba3fc3f1e84d7c8254a77f05fd82df"
format_stamp: "Formatted at 2026-09-25 15:16:31 on dist-test-slave-p468"
server_key: "841dbc33588ead37eb9b219a185bc313"
server_key_iv: "df420a438caa3a76f6bfff89b1e9ac9a"
server_key_version: "encryptionkey@0"
I20260925 15:16:54.595963 28514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:16:54.618304 28514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:16:54.622422 28514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:16:54.622639 28514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:16:54.623607 28514 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:16:54.624910 28514 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:16:54.625023 28514 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:54.625147 28514 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:16:54.625233 28514 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:54.658023 28514 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.2:45177
I20260925 15:16:54.658109 28641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.2:45177 every 8 connection(s)
I20260925 15:16:54.659508 28514 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-1/data/info.pb
I20260925 15:16:54.663887 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 28514
I20260925 15:16:54.664719 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.3:46683
--local_ip_for_outbound_sockets=127.12.8.3
--tserver_master_addrs=127.12.8.62:36947
--webserver_port=34497
--webserver_interface=127.12.8.3
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.3.keytab
--principal=kudu/127.12.8.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:38153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 15:16:54 dist-test-slave-p468 krb5kdc[27741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349414, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.2@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
I20260925 15:16:54.682950 28512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:54.662082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.2:36609 (local address 127.12.8.62:36947)
0925 15:16:54.662315 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:54.662320 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:54.663367 (+ 1047us) server_negotiation.cc:407] Connection header received
0925 15:16:54.666224 (+ 2857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:54.666232 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:54.666315 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:54.666412 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:54.668410 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:54.669105 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:54.671593 (+ 2488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:54.672014 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:54.672122 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:54.675940 (+ 3818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:16:54.675948 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:16:54.675979 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:16:54.676025 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:16:54.679576 (+ 3551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:54.680312 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:54.680320 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:54.680327 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:54.680489 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:54.680892 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:54.680900 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:54.680904 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:54.681301 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:16:54.681466 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:54.682171 (+ 705us) server_negotiation.cc:299] Negotiation successful
0925 15:16:54.682478 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 15:16:54.683597 28644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:54.662549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:36947 (local address 127.12.8.2:36609)
0925 15:16:54.663217 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:54.663258 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:54.665875 (+ 2617us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:54.666652 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:54.666665 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:54.667439 (+ 774us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:16:54.668224 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:54.668250 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:54.669255 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:54.669263 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:54.671440 (+ 2177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:54.671457 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:54.672232 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:54.672241 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:54.672565 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:54.673319 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:16:54.673341 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:16:54.675712 (+ 2371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:16:54.679726 (+ 4014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:16:54.679734 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:16:54.679746 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:16:54.680173 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:16:54.680592 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:16:54.680596 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:16:54.680598 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:16:54.680768 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:16:54.681424 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:16:54.681430 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:16:54.681799 (+ 369us) client_negotiation.cc:769] Sending connection context
0925 15:16:54.682018 (+ 219us) client_negotiation.cc:240] Negotiation successful
0925 15:16:54.682473 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":267,"threads_started":1}
I20260925 15:16:54.685148 28642 heartbeater.cc:347] Connected to a master server at 127.12.8.62:36947
I20260925 15:16:54.685451 28642 heartbeater.cc:464] Registering TS with master...
I20260925 15:16:54.686136 28642 heartbeater.cc:511] Master 127.12.8.62:36947 requested a full tablet report, sending...
I20260925 15:16:54.687589 28325 ts_manager.cc:194] Registered new tserver with Master: 7bba3fc3f1e84d7c8254a77f05fd82df (127.12.8.2:45177)
I20260925 15:16:54.688515 28325 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.2@KRBTEST.COM'} at 127.12.8.2:36609
W20260925 15:16:55.002972 28645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:16:55.003284 28645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:16:55.003455 28645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:16:55.010607 28645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:16:55.010739 28645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:16:55.010951 28645 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.3
I20260925 15:16:55.020589 28645 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:38153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.3.keytab
--principal=kudu/127.12.8.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.3:46683
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.8.3
--webserver_port=34497
--tserver_master_addrs=127.12.8.62:36947
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.3
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:16:55.021879 28645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:16:55.023307 28645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:16:55.035771 28650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:16:55.035817 28651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:16:55.035818 28653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:16:55.036484 28645 server_base.cc:1034] running on GCE node
I20260925 15:16:55.036993 28645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:16:55.037789 28645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:16:55.038978 28645 hybrid_clock.cc:648] HybridClock initialized: now 1790349415038917 us; error 53 us; skew 500 ppm
I20260925 15:16:55.039273 28645 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:16:55 dist-test-slave-p468 krb5kdc[27741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349415, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:16:55.047950 28645 init.cc:377] Logged in from keytab as kudu/127.12.8.3@KRBTEST.COM (short username kudu)
I20260925 15:16:55.049389 28645 webserver.cc:460] Webserver started at http://127.12.8.3:34497/ using document root <none> and password file <none>
I20260925 15:16:55.050042 28645 fs_manager.cc:359] Metadata directory not provided
I20260925 15:16:55.050153 28645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:16:55.055361 28645 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 15:16:55.057946 28660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:55.059280 28645 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 15:16:55.059448 28645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-2/wal
uuid: "a2cf2ac0a7a14b00986fab5569735fb8"
format_stamp: "Formatted at 2026-09-25 15:16:32 on dist-test-slave-p468"
server_key: "49a90e388af98b837903ea5dd77191a9"
server_key_iv: "3f16fb4f69de85b6124e5ac0e382c389"
server_key_version: "encryptionkey@0"
I20260925 15:16:55.060518 28645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:16:55.073943 28645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:16:55.077682 28645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:16:55.077957 28645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:16:55.078912 28645 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:16:55.080260 28645 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:16:55.080456 28645 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:55.080603 28645 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:16:55.080694 28645 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:16:55.126475 28645 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.3:46683
I20260925 15:16:55.126564 28772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.3:46683 every 8 connection(s)
I20260925 15:16:55.128264 28645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-2/data/info.pb
I20260925 15:16:55.131853 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 28645
Sep 25 15:16:55 dist-test-slave-p468 krb5kdc[27741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349415, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.3@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
I20260925 15:16:55.149564 28777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:55.133189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37414 (local address 127.12.8.62:36947)
0925 15:16:55.133670 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:55.133679 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:55.133698 (+ 19us) server_negotiation.cc:407] Connection header received
0925 15:16:55.133764 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:55.133770 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:55.133847 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:55.133969 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:55.135580 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:55.136881 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:55.138876 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:55.139659 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:55.139803 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:55.143331 (+ 3528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:16:55.143337 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:16:55.143344 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:16:55.143387 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:16:55.146839 (+ 3452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:55.147705 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:55.147709 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:55.147711 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:55.147843 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:55.148231 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:55.148234 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:55.148237 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:55.148492 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:16:55.148679 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:55.149085 (+ 406us) server_negotiation.cc:299] Negotiation successful
0925 15:16:55.149233 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":270,"threads_started":1}
I20260925 15:16:55.151950 28512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:55.131382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.3:36205 (local address 127.12.8.62:36947)
0925 15:16:55.131578 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:55.131585 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:55.132912 (+ 1327us) server_negotiation.cc:407] Connection header received
0925 15:16:55.135238 (+ 2326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:55.135242 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:55.135316 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:55.135427 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:55.137395 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:55.138523 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:55.140820 (+ 2297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:55.141575 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:55.141680 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:55.145530 (+ 3850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:16:55.145536 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:16:55.145540 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:16:55.145593 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:16:55.148578 (+ 2985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:55.149497 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:55.149502 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:55.149505 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:55.149657 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:55.150113 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:55.150118 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:55.150122 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:55.150632 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:16:55.150777 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:55.151396 (+ 619us) server_negotiation.cc:299] Negotiation successful
0925 15:16:55.151594 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 15:16:55.152808 28775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:55.131840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:36947 (local address 127.12.8.3:36205)
0925 15:16:55.132743 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:55.132788 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:55.134982 (+ 2194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:55.135675 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:55.135686 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:55.136404 (+ 718us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:16:55.137202 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:55.137226 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:55.138734 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:55.138739 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:55.140649 (+ 1910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:55.140664 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:55.142026 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:55.142034 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:55.142378 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:55.143078 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:16:55.143107 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:16:55.145336 (+ 2229us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:16:55.148741 (+ 3405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:16:55.148750 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:16:55.148765 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:16:55.149341 (+ 576us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:16:55.149820 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:16:55.149822 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:16:55.149825 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:16:55.149991 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:16:55.150752 (+ 761us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:16:55.150760 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:16:55.151055 (+ 295us) client_negotiation.cc:769] Sending connection context
0925 15:16:55.151303 (+ 248us) client_negotiation.cc:240] Negotiation successful
0925 15:16:55.151794 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":379,"threads_started":1}
I20260925 15:16:55.154057 28773 heartbeater.cc:347] Connected to a master server at 127.12.8.62:36947
I20260925 15:16:55.154372 28773 heartbeater.cc:464] Registering TS with master...
I20260925 15:16:55.155109 28773 heartbeater.cc:511] Master 127.12.8.62:36947 requested a full tablet report, sending...
I20260925 15:16:55.156584 28325 ts_manager.cc:194] Registered new tserver with Master: a2cf2ac0a7a14b00986fab5569735fb8 (127.12.8.3:46683)
I20260925 15:16:55.157407 28325 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.3@KRBTEST.COM'} at 127.12.8.3:36205
I20260925 15:16:55.164168 12320 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:16:55.170850 28512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:55.165152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37426 (local address 127.12.8.62:36947)
0925 15:16:55.165366 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:55.165374 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:55.165770 (+ 396us) server_negotiation.cc:407] Connection header received
0925 15:16:55.166033 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:55.166037 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:55.166157 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:55.166339 (+ 182us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:16:55.166766 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:55.167505 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:55.168973 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:55.169363 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:55.169493 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:55.170017 (+ 524us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:16:55.170249 (+ 232us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0925 15:16:55.170596 (+ 347us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37426: token expired
Metrics: {"server-negotiator.queue_time_us":64}
W20260925 15:16:55.170984 28512 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37426: token expired
W20260925 15:16:55.171438 28248 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.12.8.62:36947)
I20260925 15:16:55.183491 28512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:55.171899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37440 (local address 127.12.8.62:36947)
0925 15:16:55.172104 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:55.172109 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:55.172129 (+ 20us) server_negotiation.cc:407] Connection header received
0925 15:16:55.172165 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:55.172168 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:55.172224 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:55.172295 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:55.173633 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:55.174380 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:55.175588 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:55.175991 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:55.176077 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:55.178656 (+ 2579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:16:55.178661 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:16:55.178664 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:16:55.178696 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:16:55.181297 (+ 2601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:55.181924 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:55.181928 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:55.181931 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:55.182047 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:55.182402 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:55.182406 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:55.182408 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:55.182715 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:16:55.182951 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:55.183106 (+ 155us) server_negotiation.cc:299] Negotiation successful
0925 15:16:55.183267 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260925 15:16:55.186792 28325 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37440:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:16:55.194139 28325 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:16:55.219188 28786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:55.207252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54932 (local address 127.12.8.2:45177)
0925 15:16:55.208247 (+ 995us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:55.208254 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:55.208276 (+ 22us) server_negotiation.cc:407] Connection header received
0925 15:16:55.208933 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:55.208941 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:55.209189 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:55.209351 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:16:55.211463 (+ 2112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:55.213093 (+ 1630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:55.216389 (+ 3296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:55.217805 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:55.217948 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:55.218148 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:55.218500 (+ 352us) server_negotiation.cc:299] Negotiation successful
0925 15:16:55.218690 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":839,"thread_start_us":1042,"threads_started":1}
I20260925 15:16:55.219201 28785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:55.206567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.2:45177 (local address 127.0.0.1:54932)
0925 15:16:55.207542 (+ 975us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:55.207624 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:55.208589 (+ 965us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:55.209642 (+ 1053us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:55.209649 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:55.209732 (+ 83us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:16:55.211237 (+ 1505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:55.211253 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:55.213251 (+ 1998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:55.213256 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:55.216225 (+ 2969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:55.216237 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:55.218033 (+ 1796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:55.218039 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:55.218241 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:55.218306 (+ 65us) client_negotiation.cc:769] Sending connection context
0925 15:16:55.218432 (+ 126us) client_negotiation.cc:240] Negotiation successful
0925 15:16:55.218626 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":833,"mutex_wait_us":3,"spinlock_wait_cycles":84224,"thread_start_us":982,"threads_started":1}
I20260925 15:16:55.219694 28784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:55.206245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54394 (local address 127.12.8.3:46683)
0925 15:16:55.207096 (+ 851us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:55.207108 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:55.208113 (+ 1005us) server_negotiation.cc:407] Connection header received
0925 15:16:55.208980 (+ 867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:55.208987 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:55.209180 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:55.209344 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:16:55.211347 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:55.212974 (+ 1627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:55.216757 (+ 3783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:55.218163 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:55.218311 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:55.218568 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:55.219174 (+ 606us) server_negotiation.cc:299] Negotiation successful
0925 15:16:55.219346 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":692,"thread_start_us":542,"threads_started":1}
I20260925 15:16:55.220595 28788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:55.207938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.1:43327 (local address 127.0.0.1:54730)
0925 15:16:55.210862 (+ 2924us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:55.210878 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:55.211389 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:55.212239 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:55.212243 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:55.212281 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:16:55.212622 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:55.212639 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:55.214716 (+ 2077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:55.214721 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:55.216952 (+ 2231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:55.216961 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:55.219695 (+ 2734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:55.219700 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:55.219877 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:55.219906 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 15:16:55.219996 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 15:16:55.220124 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2784,"mutex_wait_us":41,"thread_start_us":2642,"threads_started":1}
I20260925 15:16:55.220628 28783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:55.205880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.3:46683 (local address 127.0.0.1:54394)
0925 15:16:55.207542 (+ 1662us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:55.208020 (+ 478us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:55.208801 (+ 781us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:55.209642 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:55.209649 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:55.209726 (+ 77us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:16:55.211173 (+ 1447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:55.211189 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:55.213173 (+ 1984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:55.213181 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:55.216601 (+ 3420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:55.216616 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:55.218729 (+ 2113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:55.218735 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:55.218950 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:55.218996 (+ 46us) client_negotiation.cc:769] Sending connection context
0925 15:16:55.219948 (+ 952us) client_negotiation.cc:240] Negotiation successful
0925 15:16:55.220086 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1377,"mutex_wait_us":63,"spinlock_wait_cycles":1224064,"thread_start_us":1206,"threads_started":1}
I20260925 15:16:55.222975 28577 tablet_service.cc:1467] Processing CreateTablet for tablet 086b98a4cabd40c0bc4198768dba2620 (DEFAULT_TABLE table=test-table [id=47b85734fa5f4302aeb07b66318c3534]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:16:55.224330 28577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 086b98a4cabd40c0bc4198768dba2620. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:16:55.223920 28708 tablet_service.cc:1467] Processing CreateTablet for tablet 086b98a4cabd40c0bc4198768dba2620 (DEFAULT_TABLE table=test-table [id=47b85734fa5f4302aeb07b66318c3534]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:16:55.225096 28708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 086b98a4cabd40c0bc4198768dba2620. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:16:55.228550 28787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:55.208271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54730 (local address 127.12.8.1:43327)
0925 15:16:55.208789 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:55.208796 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:55.210983 (+ 2187us) server_negotiation.cc:407] Connection header received
0925 15:16:55.211595 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:55.211602 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:55.211838 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:55.211987 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:16:55.212788 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:55.214531 (+ 1743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:55.217085 (+ 2554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:55.218283 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:55.218985 (+ 702us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:55.219199 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:55.220037 (+ 838us) server_negotiation.cc:299] Negotiation successful
0925 15:16:55.220192 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":296,"threads_started":1}
I20260925 15:16:55.231732 28445 tablet_service.cc:1467] Processing CreateTablet for tablet 086b98a4cabd40c0bc4198768dba2620 (DEFAULT_TABLE table=test-table [id=47b85734fa5f4302aeb07b66318c3534]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:16:55.232833 28445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 086b98a4cabd40c0bc4198768dba2620. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:16:55.239002 28789 tablet_bootstrap.cc:492] T 086b98a4cabd40c0bc4198768dba2620 P 7bba3fc3f1e84d7c8254a77f05fd82df: Bootstrap starting.
I20260925 15:16:55.241051 28790 tablet_bootstrap.cc:492] T 086b98a4cabd40c0bc4198768dba2620 P a2cf2ac0a7a14b00986fab5569735fb8: Bootstrap starting.
I20260925 15:16:55.242677 28789 tablet_bootstrap.cc:654] T 086b98a4cabd40c0bc4198768dba2620 P 7bba3fc3f1e84d7c8254a77f05fd82df: Neither blocks nor log segments found. Creating new log.
I20260925 15:16:55.244065 28789 log.cc:824] T 086b98a4cabd40c0bc4198768dba2620 P 7bba3fc3f1e84d7c8254a77f05fd82df: Log is configured to *not* fsync() on all Append() calls
I20260925 15:16:55.244210 28790 tablet_bootstrap.cc:654] T 086b98a4cabd40c0bc4198768dba2620 P a2cf2ac0a7a14b00986fab5569735fb8: Neither blocks nor log segments found. Creating new log.
I20260925 15:16:55.245441 28790 log.cc:824] T 086b98a4cabd40c0bc4198768dba2620 P a2cf2ac0a7a14b00986fab5569735fb8: Log is configured to *not* fsync() on all Append() calls
I20260925 15:16:55.247916 28790 tablet_bootstrap.cc:492] T 086b98a4cabd40c0bc4198768dba2620 P a2cf2ac0a7a14b00986fab5569735fb8: No bootstrap required, opened a new log
I20260925 15:16:55.248100 28789 tablet_bootstrap.cc:492] T 086b98a4cabd40c0bc4198768dba2620 P 7bba3fc3f1e84d7c8254a77f05fd82df: No bootstrap required, opened a new log
I20260925 15:16:55.248185 28790 ts_tablet_manager.cc:1397] T 086b98a4cabd40c0bc4198768dba2620 P a2cf2ac0a7a14b00986fab5569735fb8: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260925 15:16:55.248407 28789 ts_tablet_manager.cc:1397] T 086b98a4cabd40c0bc4198768dba2620 P 7bba3fc3f1e84d7c8254a77f05fd82df: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260925 15:16:55.251158 28510 heartbeater.cc:503] Master 127.12.8.62:36947 was elected leader, sending a full tablet report...
I20260925 15:16:55.251917 28792 tablet_bootstrap.cc:492] T 086b98a4cabd40c0bc4198768dba2620 P 232b8a44ea8e4bd39da1eed7fcb322e4: Bootstrap starting.
I20260925 15:16:55.255368 28792 tablet_bootstrap.cc:654] T 086b98a4cabd40c0bc4198768dba2620 P 232b8a44ea8e4bd39da1eed7fcb322e4: Neither blocks nor log segments found. Creating new log.
I20260925 15:16:55.256940 28790 raft_consensus.cc:348] T 086b98a4cabd40c0bc4198768dba2620 P a2cf2ac0a7a14b00986fab5569735fb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2cf2ac0a7a14b00986fab5569735fb8" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 46683 } } peers { permanent_uuid: "232b8a44ea8e4bd39da1eed7fcb322e4" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 43327 } } peers { permanent_uuid: "7bba3fc3f1e84d7c8254a77f05fd82df" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 45177 } }
I20260925 15:16:55.257370 28790 raft_consensus.cc:374] T 086b98a4cabd40c0bc4198768dba2620 P a2cf2ac0a7a14b00986fab5569735fb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:16:55.257480 28790 raft_consensus.cc:729] T 086b98a4cabd40c0bc4198768dba2620 P a2cf2ac0a7a14b00986fab5569735fb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2cf2ac0a7a14b00986fab5569735fb8, State: Initialized, Role: FOLLOWER
I20260925 15:16:55.257481 28792 log.cc:824] T 086b98a4cabd40c0bc4198768dba2620 P 232b8a44ea8e4bd39da1eed7fcb322e4: Log is configured to *not* fsync() on all Append() calls
I20260925 15:16:55.258113 28790 consensus_queue.cc:260] T 086b98a4cabd40c0bc4198768dba2620 P a2cf2ac0a7a14b00986fab5569735fb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2cf2ac0a7a14b00986fab5569735fb8" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 46683 } } peers { permanent_uuid: "232b8a44ea8e4bd39da1eed7fcb322e4" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 43327 } } peers { permanent_uuid: "7bba3fc3f1e84d7c8254a77f05fd82df" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 45177 } }
I20260925 15:16:55.259545 28790 ts_tablet_manager.cc:1428] T 086b98a4cabd40c0bc4198768dba2620 P a2cf2ac0a7a14b00986fab5569735fb8: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260925 15:16:55.259966 28773 heartbeater.cc:503] Master 127.12.8.62:36947 was elected leader, sending a full tablet report...
I20260925 15:16:55.259756 28789 raft_consensus.cc:348] T 086b98a4cabd40c0bc4198768dba2620 P 7bba3fc3f1e84d7c8254a77f05fd82df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2cf2ac0a7a14b00986fab5569735fb8" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 46683 } } peers { permanent_uuid: "232b8a44ea8e4bd39da1eed7fcb322e4" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 43327 } } peers { permanent_uuid: "7bba3fc3f1e84d7c8254a77f05fd82df" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 45177 } }
I20260925 15:16:55.260181 28789 raft_consensus.cc:374] T 086b98a4cabd40c0bc4198768dba2620 P 7bba3fc3f1e84d7c8254a77f05fd82df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:16:55.260288 28789 raft_consensus.cc:729] T 086b98a4cabd40c0bc4198768dba2620 P 7bba3fc3f1e84d7c8254a77f05fd82df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bba3fc3f1e84d7c8254a77f05fd82df, State: Initialized, Role: FOLLOWER
I20260925 15:16:55.260538 28792 tablet_bootstrap.cc:492] T 086b98a4cabd40c0bc4198768dba2620 P 232b8a44ea8e4bd39da1eed7fcb322e4: No bootstrap required, opened a new log
I20260925 15:16:55.260779 28792 ts_tablet_manager.cc:1397] T 086b98a4cabd40c0bc4198768dba2620 P 232b8a44ea8e4bd39da1eed7fcb322e4: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260925 15:16:55.260842 28789 consensus_queue.cc:260] T 086b98a4cabd40c0bc4198768dba2620 P 7bba3fc3f1e84d7c8254a77f05fd82df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2cf2ac0a7a14b00986fab5569735fb8" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 46683 } } peers { permanent_uuid: "232b8a44ea8e4bd39da1eed7fcb322e4" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 43327 } } peers { permanent_uuid: "7bba3fc3f1e84d7c8254a77f05fd82df" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 45177 } }
I20260925 15:16:55.261905 28642 heartbeater.cc:503] Master 127.12.8.62:36947 was elected leader, sending a full tablet report...
I20260925 15:16:55.262531 28789 ts_tablet_manager.cc:1428] T 086b98a4cabd40c0bc4198768dba2620 P 7bba3fc3f1e84d7c8254a77f05fd82df: Time spent starting tablet: real 0.014s user 0.016s sys 0.000s
I20260925 15:16:55.269400 28792 raft_consensus.cc:348] T 086b98a4cabd40c0bc4198768dba2620 P 232b8a44ea8e4bd39da1eed7fcb322e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2cf2ac0a7a14b00986fab5569735fb8" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 46683 } } peers { permanent_uuid: "232b8a44ea8e4bd39da1eed7fcb322e4" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 43327 } } peers { permanent_uuid: "7bba3fc3f1e84d7c8254a77f05fd82df" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 45177 } }
I20260925 15:16:55.269704 28792 raft_consensus.cc:374] T 086b98a4cabd40c0bc4198768dba2620 P 232b8a44ea8e4bd39da1eed7fcb322e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:16:55.269775 28792 raft_consensus.cc:729] T 086b98a4cabd40c0bc4198768dba2620 P 232b8a44ea8e4bd39da1eed7fcb322e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 232b8a44ea8e4bd39da1eed7fcb322e4, State: Initialized, Role: FOLLOWER
I20260925 15:16:55.270169 28792 consensus_queue.cc:260] T 086b98a4cabd40c0bc4198768dba2620 P 232b8a44ea8e4bd39da1eed7fcb322e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2cf2ac0a7a14b00986fab5569735fb8" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 46683 } } peers { permanent_uuid: "232b8a44ea8e4bd39da1eed7fcb322e4" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 43327 } } peers { permanent_uuid: "7bba3fc3f1e84d7c8254a77f05fd82df" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 45177 } }
I20260925 15:16:55.271402 28792 ts_tablet_manager.cc:1428] T 086b98a4cabd40c0bc4198768dba2620 P 232b8a44ea8e4bd39da1eed7fcb322e4: Time spent starting tablet: real 0.011s user 0.003s sys 0.009s
W20260925 15:16:55.380084 28774 tablet.cc:2369] T 086b98a4cabd40c0bc4198768dba2620 P a2cf2ac0a7a14b00986fab5569735fb8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:16:55.411509 28643 tablet.cc:2369] T 086b98a4cabd40c0bc4198768dba2620 P 7bba3fc3f1e84d7c8254a77f05fd82df: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:16:55.462471 28511 tablet.cc:2369] T 086b98a4cabd40c0bc4198768dba2620 P 232b8a44ea8e4bd39da1eed7fcb322e4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:16:55.497284 28796 raft_consensus.cc:482] T 086b98a4cabd40c0bc4198768dba2620 P 7bba3fc3f1e84d7c8254a77f05fd82df [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:16:55.497555 28796 raft_consensus.cc:504] T 086b98a4cabd40c0bc4198768dba2620 P 7bba3fc3f1e84d7c8254a77f05fd82df [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2cf2ac0a7a14b00986fab5569735fb8" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 46683 } } peers { permanent_uuid: "232b8a44ea8e4bd39da1eed7fcb322e4" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 43327 } } peers { permanent_uuid: "7bba3fc3f1e84d7c8254a77f05fd82df" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 45177 } }
I20260925 15:16:55.499567 28796 leader_election.cc:288] T 086b98a4cabd40c0bc4198768dba2620 P 7bba3fc3f1e84d7c8254a77f05fd82df [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2cf2ac0a7a14b00986fab5569735fb8 (127.12.8.3:46683), 232b8a44ea8e4bd39da1eed7fcb322e4 (127.12.8.1:43327)
I20260925 15:16:55.506829 28784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:55.500025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.2:59507 (local address 127.12.8.3:46683)
0925 15:16:55.500233 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:55.500241 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:55.500762 (+ 521us) server_negotiation.cc:407] Connection header received
0925 15:16:55.500995 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:55.501001 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:55.501092 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:55.501159 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:16:55.502061 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:55.502890 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:55.505012 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:55.505859 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:55.505961 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:55.506081 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:55.506425 (+ 344us) server_negotiation.cc:299] Negotiation successful
0925 15:16:55.506570 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 15:16:55.506917 28798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:55.499946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.3:46683 (local address 127.12.8.2:59507)
0925 15:16:55.500614 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:55.500650 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:55.500810 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:55.501368 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:55.501374 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:55.501478 (+ 104us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:16:55.501908 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:55.501919 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:55.503021 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:55.503026 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:55.504872 (+ 1846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:55.504884 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:55.506078 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:55.506085 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:55.506281 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:55.506304 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 15:16:55.506382 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 15:16:55.506467 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"mutex_wait_us":47,"spinlock_wait_cycles":25984,"thread_start_us":393,"threads_started":1}
I20260925 15:16:55.506945 28787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:55.500955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.2:49227 (local address 127.12.8.1:43327)
0925 15:16:55.501153 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:55.501159 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:55.501399 (+ 240us) server_negotiation.cc:407] Connection header received
0925 15:16:55.501513 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:55.501520 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:55.501611 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:55.501708 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:16:55.502348 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:55.503103 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:55.505152 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:55.505906 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:55.506065 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:55.506162 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:55.506576 (+ 414us) server_negotiation.cc:299] Negotiation successful
0925 15:16:55.506710 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 15:16:55.506989 28799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:55.500690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.1:43327 (local address 127.12.8.2:49227)
0925 15:16:55.501296 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0925 15:16:55.501309 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:16:55.501410 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:16:55.501730 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:16:55.501733 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:16:55.501750 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:16:55.502155 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:55.502171 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:55.503258 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:55.503265 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:55.505007 (+ 1742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:16:55.505021 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:16:55.506155 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:16:55.506165 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:16:55.506399 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:55.506431 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 15:16:55.506523 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 15:16:55.506640 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":312,"threads_started":1}
I20260925 15:16:55.508095 28465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086b98a4cabd40c0bc4198768dba2620" candidate_uuid: "7bba3fc3f1e84d7c8254a77f05fd82df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "232b8a44ea8e4bd39da1eed7fcb322e4" is_pre_election: true
I20260925 15:16:55.508131 28728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086b98a4cabd40c0bc4198768dba2620" candidate_uuid: "7bba3fc3f1e84d7c8254a77f05fd82df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2cf2ac0a7a14b00986fab5569735fb8" is_pre_election: true
I20260925 15:16:55.508601 28728 raft_consensus.cc:2454] T 086b98a4cabd40c0bc4198768dba2620 P a2cf2ac0a7a14b00986fab5569735fb8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7bba3fc3f1e84d7c8254a77f05fd82df in term 0.
I20260925 15:16:55.508610 28465 raft_consensus.cc:2454] T 086b98a4cabd40c0bc4198768dba2620 P 232b8a44ea8e4bd39da1eed7fcb322e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7bba3fc3f1e84d7c8254a77f05fd82df in term 0.
I20260925 15:16:55.509244 28530 leader_election.cc:302] T 086b98a4cabd40c0bc4198768dba2620 P 7bba3fc3f1e84d7c8254a77f05fd82df [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7bba3fc3f1e84d7c8254a77f05fd82df, a2cf2ac0a7a14b00986fab5569735fb8; no voters:
I20260925 15:16:55.509622 28796 raft_consensus.cc:2781] T 086b98a4cabd40c0bc4198768dba2620 P 7bba3fc3f1e84d7c8254a77f05fd82df [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:16:55.509778 28796 raft_consensus.cc:482] T 086b98a4cabd40c0bc4198768dba2620 P 7bba3fc3f1e84d7c8254a77f05fd82df [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:16:55.509902 28796 raft_consensus.cc:3037] T 086b98a4cabd40c0bc4198768dba2620 P 7bba3fc3f1e84d7c8254a77f05fd82df [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:16:55.512132 28796 raft_consensus.cc:504] T 086b98a4cabd40c0bc4198768dba2620 P 7bba3fc3f1e84d7c8254a77f05fd82df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2cf2ac0a7a14b00986fab5569735fb8" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 46683 } } peers { permanent_uuid: "232b8a44ea8e4bd39da1eed7fcb322e4" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 43327 } } peers { permanent_uuid: "7bba3fc3f1e84d7c8254a77f05fd82df" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 45177 } }
I20260925 15:16:55.512681 28796 leader_election.cc:288] T 086b98a4cabd40c0bc4198768dba2620 P 7bba3fc3f1e84d7c8254a77f05fd82df [CANDIDATE]: Term 1 election: Requested vote from peers a2cf2ac0a7a14b00986fab5569735fb8 (127.12.8.3:46683), 232b8a44ea8e4bd39da1eed7fcb322e4 (127.12.8.1:43327)
I20260925 15:16:55.513108 28728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086b98a4cabd40c0bc4198768dba2620" candidate_uuid: "7bba3fc3f1e84d7c8254a77f05fd82df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2cf2ac0a7a14b00986fab5569735fb8"
I20260925 15:16:55.513329 28728 raft_consensus.cc:3037] T 086b98a4cabd40c0bc4198768dba2620 P a2cf2ac0a7a14b00986fab5569735fb8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:16:55.513300 28465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086b98a4cabd40c0bc4198768dba2620" candidate_uuid: "7bba3fc3f1e84d7c8254a77f05fd82df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "232b8a44ea8e4bd39da1eed7fcb322e4"
I20260925 15:16:55.513499 28465 raft_consensus.cc:3037] T 086b98a4cabd40c0bc4198768dba2620 P 232b8a44ea8e4bd39da1eed7fcb322e4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:16:55.516547 28728 raft_consensus.cc:2454] T 086b98a4cabd40c0bc4198768dba2620 P a2cf2ac0a7a14b00986fab5569735fb8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7bba3fc3f1e84d7c8254a77f05fd82df in term 1.
I20260925 15:16:55.516623 28465 raft_consensus.cc:2454] T 086b98a4cabd40c0bc4198768dba2620 P 232b8a44ea8e4bd39da1eed7fcb322e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7bba3fc3f1e84d7c8254a77f05fd82df in term 1.
I20260925 15:16:55.516942 28530 leader_election.cc:302] T 086b98a4cabd40c0bc4198768dba2620 P 7bba3fc3f1e84d7c8254a77f05fd82df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7bba3fc3f1e84d7c8254a77f05fd82df, a2cf2ac0a7a14b00986fab5569735fb8; no voters:
I20260925 15:16:55.517256 28796 raft_consensus.cc:2781] T 086b98a4cabd40c0bc4198768dba2620 P 7bba3fc3f1e84d7c8254a77f05fd82df [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:16:55.517688 28796 raft_consensus.cc:686] T 086b98a4cabd40c0bc4198768dba2620 P 7bba3fc3f1e84d7c8254a77f05fd82df [term 1 LEADER]: Becoming Leader. State: Replica: 7bba3fc3f1e84d7c8254a77f05fd82df, State: Running, Role: LEADER
I20260925 15:16:55.518142 28796 consensus_queue.cc:237] T 086b98a4cabd40c0bc4198768dba2620 P 7bba3fc3f1e84d7c8254a77f05fd82df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2cf2ac0a7a14b00986fab5569735fb8" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 46683 } } peers { permanent_uuid: "232b8a44ea8e4bd39da1eed7fcb322e4" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 43327 } } peers { permanent_uuid: "7bba3fc3f1e84d7c8254a77f05fd82df" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 45177 } }
I20260925 15:16:55.522370 28325 catalog_manager.cc:5495] T 086b98a4cabd40c0bc4198768dba2620 P 7bba3fc3f1e84d7c8254a77f05fd82df reported cstate change: term changed from 0 to 1, leader changed from <none> to 7bba3fc3f1e84d7c8254a77f05fd82df (127.12.8.2). New cstate: current_term: 1 leader_uuid: "7bba3fc3f1e84d7c8254a77f05fd82df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2cf2ac0a7a14b00986fab5569735fb8" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 46683 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "232b8a44ea8e4bd39da1eed7fcb322e4" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 43327 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7bba3fc3f1e84d7c8254a77f05fd82df" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 45177 } health_report { overall_health: HEALTHY } } }
I20260925 15:16:55.581504 28786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:55.576448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54942 (local address 127.12.8.2:45177)
0925 15:16:55.576675 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:55.576680 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:55.576694 (+ 14us) server_negotiation.cc:407] Connection header received
0925 15:16:55.576771 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:55.576775 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:55.576856 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:55.577005 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:16:55.577365 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:55.578052 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:55.579428 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:55.579831 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:55.579960 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:55.580443 (+ 483us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:16:55.580914 (+ 471us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:16:55.581071 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:55.581160 (+ 89us) server_negotiation.cc:299] Negotiation successful
0925 15:16:55.581236 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 15:16:55.589285 28465 raft_consensus.cc:1261] T 086b98a4cabd40c0bc4198768dba2620 P 232b8a44ea8e4bd39da1eed7fcb322e4 [term 1 FOLLOWER]: Refusing update from remote peer 7bba3fc3f1e84d7c8254a77f05fd82df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:16:55.589294 28728 raft_consensus.cc:1261] T 086b98a4cabd40c0bc4198768dba2620 P a2cf2ac0a7a14b00986fab5569735fb8 [term 1 FOLLOWER]: Refusing update from remote peer 7bba3fc3f1e84d7c8254a77f05fd82df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:16:55.590252 28800 consensus_queue.cc:1035] T 086b98a4cabd40c0bc4198768dba2620 P 7bba3fc3f1e84d7c8254a77f05fd82df [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2cf2ac0a7a14b00986fab5569735fb8" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 46683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:16:55.590633 28796 consensus_queue.cc:1035] T 086b98a4cabd40c0bc4198768dba2620 P 7bba3fc3f1e84d7c8254a77f05fd82df [LEADER]: Connected to new peer: Peer: permanent_uuid: "232b8a44ea8e4bd39da1eed7fcb322e4" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 43327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:16:55.615144 28324 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37440:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:16:55.615406 28324 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37440:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:16:55.618777 28324 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 46041bb901664061b44eb699831c4e55: Sending DeleteTablet for 3 replicas of tablet 086b98a4cabd40c0bc4198768dba2620
I20260925 15:16:55.619858 28708 tablet_service.cc:1514] Processing DeleteTablet for tablet 086b98a4cabd40c0bc4198768dba2620 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:16:55 UTC) from {username='kudu', principal='kudu/127.12.8.62@KRBTEST.COM'} at 127.0.0.1:54394
I20260925 15:16:55.619899 28445 tablet_service.cc:1514] Processing DeleteTablet for tablet 086b98a4cabd40c0bc4198768dba2620 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:16:55 UTC) from {username='kudu', principal='kudu/127.12.8.62@KRBTEST.COM'} at 127.0.0.1:54730
I20260925 15:16:55.619906 28577 tablet_service.cc:1514] Processing DeleteTablet for tablet 086b98a4cabd40c0bc4198768dba2620 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:16:55 UTC) from {username='kudu', principal='kudu/127.12.8.62@KRBTEST.COM'} at 127.0.0.1:54932
I20260925 15:16:55.624688 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 28362
I20260925 15:16:55.633021 28815 tablet_replica.cc:330] T 086b98a4cabd40c0bc4198768dba2620 P 7bba3fc3f1e84d7c8254a77f05fd82df: stopping tablet replica
I20260925 15:16:55.633687 28815 raft_consensus.cc:2229] T 086b98a4cabd40c0bc4198768dba2620 P 7bba3fc3f1e84d7c8254a77f05fd82df [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:16:55.634743 28813 tablet_replica.cc:330] T 086b98a4cabd40c0bc4198768dba2620 P a2cf2ac0a7a14b00986fab5569735fb8: stopping tablet replica
I20260925 15:16:55.635215 28813 raft_consensus.cc:2229] T 086b98a4cabd40c0bc4198768dba2620 P a2cf2ac0a7a14b00986fab5569735fb8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:16:55.635808 28813 raft_consensus.cc:2258] T 086b98a4cabd40c0bc4198768dba2620 P a2cf2ac0a7a14b00986fab5569735fb8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:16:55.636529 28815 raft_consensus.cc:2258] T 086b98a4cabd40c0bc4198768dba2620 P 7bba3fc3f1e84d7c8254a77f05fd82df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:16:55.637123 28814 tablet_replica.cc:330] T 086b98a4cabd40c0bc4198768dba2620 P 232b8a44ea8e4bd39da1eed7fcb322e4: stopping tablet replica
I20260925 15:16:55.637537 28814 raft_consensus.cc:2229] T 086b98a4cabd40c0bc4198768dba2620 P 232b8a44ea8e4bd39da1eed7fcb322e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:16:55.638073 28814 raft_consensus.cc:2258] T 086b98a4cabd40c0bc4198768dba2620 P 232b8a44ea8e4bd39da1eed7fcb322e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:16:55.638792 28815 ts_tablet_manager.cc:1905] T 086b98a4cabd40c0bc4198768dba2620 P 7bba3fc3f1e84d7c8254a77f05fd82df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:16:55.640244 28814 ts_tablet_manager.cc:1905] T 086b98a4cabd40c0bc4198768dba2620 P 232b8a44ea8e4bd39da1eed7fcb322e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:16:55.646258 28815 ts_tablet_manager.cc:1918] T 086b98a4cabd40c0bc4198768dba2620 P 7bba3fc3f1e84d7c8254a77f05fd82df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:16:55.646516 28815 log.cc:1196] T 086b98a4cabd40c0bc4198768dba2620 P 7bba3fc3f1e84d7c8254a77f05fd82df: Deleting WAL directory at /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-1/wal/wals/086b98a4cabd40c0bc4198768dba2620
I20260925 15:16:55.647123 28815 ts_tablet_manager.cc:1939] T 086b98a4cabd40c0bc4198768dba2620 P 7bba3fc3f1e84d7c8254a77f05fd82df: Deleting consensus metadata
I20260925 15:16:55.647332 28814 ts_tablet_manager.cc:1918] T 086b98a4cabd40c0bc4198768dba2620 P 232b8a44ea8e4bd39da1eed7fcb322e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:16:55.647657 28814 log.cc:1196] T 086b98a4cabd40c0bc4198768dba2620 P 232b8a44ea8e4bd39da1eed7fcb322e4: Deleting WAL directory at /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-0/wal/wals/086b98a4cabd40c0bc4198768dba2620
I20260925 15:16:55.648248 28814 ts_tablet_manager.cc:1939] T 086b98a4cabd40c0bc4198768dba2620 P 232b8a44ea8e4bd39da1eed7fcb322e4: Deleting consensus metadata
I20260925 15:16:55.648358 28787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:55.626244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54734 (local address 127.12.8.1:43327)
0925 15:16:55.626405 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:55.626412 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:55.626429 (+ 17us) server_negotiation.cc:407] Connection header received
0925 15:16:55.626482 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:55.626488 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:55.626603 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:55.626707 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:55.628715 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:55.629951 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:55.631907 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:55.632563 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:55.632682 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:55.637816 (+ 5134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:16:55.637832 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:16:55.637850 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:16:55.637893 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:16:55.642307 (+ 4414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:55.643358 (+ 1051us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:55.643368 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:55.643372 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:55.643598 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:55.644114 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:55.644120 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:55.644124 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:55.644676 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:16:55.645055 (+ 379us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:55.647626 (+ 2571us) server_negotiation.cc:299] Negotiation successful
0925 15:16:55.647905 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 15:16:55.649137 28312 catalog_manager.cc:4841] TS 7bba3fc3f1e84d7c8254a77f05fd82df (127.12.8.2:45177): tablet 086b98a4cabd40c0bc4198768dba2620 (table test-table [id=47b85734fa5f4302aeb07b66318c3534]) successfully deleted
I20260925 15:16:55.649485 28505 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:16:55.649969 28309 catalog_manager.cc:4841] TS 232b8a44ea8e4bd39da1eed7fcb322e4 (127.12.8.1:43327): tablet 086b98a4cabd40c0bc4198768dba2620 (table test-table [id=47b85734fa5f4302aeb07b66318c3534]) successfully deleted
I20260925 15:16:55.650089 28813 ts_tablet_manager.cc:1905] T 086b98a4cabd40c0bc4198768dba2620 P a2cf2ac0a7a14b00986fab5569735fb8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:16:55.655253 28813 ts_tablet_manager.cc:1918] T 086b98a4cabd40c0bc4198768dba2620 P a2cf2ac0a7a14b00986fab5569735fb8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:16:55.655421 28813 log.cc:1196] T 086b98a4cabd40c0bc4198768dba2620 P a2cf2ac0a7a14b00986fab5569735fb8: Deleting WAL directory at /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-2/wal/wals/086b98a4cabd40c0bc4198768dba2620
I20260925 15:16:55.655824 28813 ts_tablet_manager.cc:1939] T 086b98a4cabd40c0bc4198768dba2620 P a2cf2ac0a7a14b00986fab5569735fb8: Deleting consensus metadata
I20260925 15:16:55.657366 28309 catalog_manager.cc:4841] TS a2cf2ac0a7a14b00986fab5569735fb8 (127.12.8.3:46683): tablet 086b98a4cabd40c0bc4198768dba2620 (table test-table [id=47b85734fa5f4302aeb07b66318c3534]) successfully deleted
I20260925 15:16:55.753088 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 28362
I20260925 15:16:55.771387 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 28514
I20260925 15:16:55.788086 28786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:55.772183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54958 (local address 127.12.8.2:45177)
0925 15:16:55.772357 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:55.772362 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:55.772473 (+ 111us) server_negotiation.cc:407] Connection header received
0925 15:16:55.772672 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:55.772676 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:55.772737 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:55.772823 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:55.774862 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:55.775645 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:55.777806 (+ 2161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:55.778340 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:55.778442 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:55.781916 (+ 3474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:16:55.781927 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:16:55.781942 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:16:55.781978 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:16:55.785020 (+ 3042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:55.785855 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:55.785861 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:55.785864 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:55.785998 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:55.786523 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:55.786526 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:55.786528 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:55.786854 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:16:55.787195 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:55.787510 (+ 315us) server_negotiation.cc:299] Negotiation successful
0925 15:16:55.787722 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 15:16:55.788802 28637 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:16:55.902464 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 28514
I20260925 15:16:55.920846 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 28645
I20260925 15:16:55.934512 28784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:16:55.921728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54398 (local address 127.12.8.3:46683)
0925 15:16:55.921926 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 15:16:55.921930 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:16:55.921971 (+ 41us) server_negotiation.cc:407] Connection header received
0925 15:16:55.922279 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:16:55.922284 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:16:55.922352 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:16:55.922442 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:16:55.923946 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:55.924713 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:55.926182 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:16:55.926619 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:16:55.926715 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:16:55.929550 (+ 2835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:16:55.929562 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:16:55.929581 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:16:55.929616 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:16:55.932114 (+ 2498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:55.932716 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:55.932722 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:55.932724 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:55.932869 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:16:55.933182 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:16:55.933185 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:16:55.933187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:16:55.933495 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:16:55.933692 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0925 15:16:55.934042 (+ 350us) server_negotiation.cc:299] Negotiation successful
0925 15:16:55.934270 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 15:16:55.935112 28768 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:16:56.024443 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 28645
I20260925 15:16:56.040935 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 28292
I20260925 15:16:56.042042 28355 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:16:56.154101 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 28292
I20260925 15:17:16.175472 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.62:36947
--webserver_interface=127.12.8.62
--webserver_port=41499
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab
--principal=kudu/127.12.8.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:38153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.62:36947
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:17:16.502712 28820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:17:16.503000 28820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:17:16.503196 28820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:17:16.510411 28820 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 15:17:16.510499 28820 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 15:17:16.510592 28820 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 15:17:16.510675 28820 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:17:16.510771 28820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:17:16.510885 28820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:17:16.510954 28820 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:17:16.511032 28820 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:17:16.520768 28820 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:38153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.12.8.62:36947
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab
--principal=kudu/127.12.8.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.62:36947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.8.62
--webserver_port=41499
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:17:16.522035 28820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:17:16.523389 28820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:17:16.535346 28825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:17:16.535399 28826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:17:16.535298 28828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:17:16.539651 28820 server_base.cc:1034] running on GCE node
I20260925 15:17:16.540484 28820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:17:16.541860 28820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:17:16.543097 28820 hybrid_clock.cc:648] HybridClock initialized: now 1790349436543038 us; error 37 us; skew 500 ppm
I20260925 15:17:16.543428 28820 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:17:16 dist-test-slave-p468 krb5kdc[27741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349436, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:17:16.546597 28820 init.cc:377] Logged in from keytab as kudu/127.12.8.62@KRBTEST.COM (short username kudu)
I20260925 15:17:16.548110 28820 webserver.cc:460] Webserver started at http://127.12.8.62:41499/ using document root <none> and password file <none>
I20260925 15:17:16.548812 28820 fs_manager.cc:359] Metadata directory not provided
I20260925 15:17:16.548941 28820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:17:16.554231 28820 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:17:16.556953 28835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:17:16.558369 28820 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:17:16.558539 28820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/master-0/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/master-0/wal
uuid: "46041bb901664061b44eb699831c4e55"
format_stamp: "Formatted at 2026-09-25 15:16:30 on dist-test-slave-p468"
server_key: "d1172c8c8c128130cd60dc56711820cc"
server_key_iv: "0884fac03bac734537c56062624c1806"
server_key_version: "encryptionkey@0"
I20260925 15:17:16.559428 28820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:17:16.583680 28820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:17:16.587549 28820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:17:16.587841 28820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:17:16.606316 28820 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.62:36947
I20260925 15:17:16.606406 28886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.62:36947 every 8 connection(s)
I20260925 15:17:16.607764 28820 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
I20260925 15:17:16.614622 28887 sys_catalog.cc:263] Verifying existing consensus state
I20260925 15:17:16.617781 28887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46041bb901664061b44eb699831c4e55: Bootstrap starting.
I20260925 15:17:16.618320 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 28820
I20260925 15:17:16.619091 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.1:43327
--local_ip_for_outbound_sockets=127.12.8.1
--tserver_master_addrs=127.12.8.62:36947
--webserver_port=45247
--webserver_interface=127.12.8.1
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.1.keytab
--principal=kudu/127.12.8.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:38153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:17:16.640884 28887 log.cc:824] T 00000000000000000000000000000000 P 46041bb901664061b44eb699831c4e55: Log is configured to *not* fsync() on all Append() calls
I20260925 15:17:16.657161 28887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46041bb901664061b44eb699831c4e55: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 15:17:16.657739 28887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46041bb901664061b44eb699831c4e55: Bootstrap complete.
I20260925 15:17:16.667090 28887 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46041bb901664061b44eb699831c4e55 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46041bb901664061b44eb699831c4e55" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 36947 } }
I20260925 15:17:16.667999 28887 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46041bb901664061b44eb699831c4e55 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46041bb901664061b44eb699831c4e55, State: Initialized, Role: FOLLOWER
I20260925 15:17:16.668602 28887 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46041bb901664061b44eb699831c4e55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46041bb901664061b44eb699831c4e55" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 36947 } }
I20260925 15:17:16.668852 28887 raft_consensus.cc:388] T 00000000000000000000000000000000 P 46041bb901664061b44eb699831c4e55 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:17:16.668990 28887 raft_consensus.cc:482] T 00000000000000000000000000000000 P 46041bb901664061b44eb699831c4e55 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:17:16.669184 28887 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46041bb901664061b44eb699831c4e55 [term 2 FOLLOWER]: Advancing to term 3
I20260925 15:17:16.672968 28887 raft_consensus.cc:504] T 00000000000000000000000000000000 P 46041bb901664061b44eb699831c4e55 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46041bb901664061b44eb699831c4e55" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 36947 } }
I20260925 15:17:16.673470 28887 leader_election.cc:302] T 00000000000000000000000000000000 P 46041bb901664061b44eb699831c4e55 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 46041bb901664061b44eb699831c4e55; no voters:
I20260925 15:17:16.674209 28887 leader_election.cc:288] T 00000000000000000000000000000000 P 46041bb901664061b44eb699831c4e55 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 15:17:16.674346 28891 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 46041bb901664061b44eb699831c4e55 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 15:17:16.675659 28891 raft_consensus.cc:686] T 00000000000000000000000000000000 P 46041bb901664061b44eb699831c4e55 [term 3 LEADER]: Becoming Leader. State: Replica: 46041bb901664061b44eb699831c4e55, State: Running, Role: LEADER
I20260925 15:17:16.676203 28891 consensus_queue.cc:237] T 00000000000000000000000000000000 P 46041bb901664061b44eb699831c4e55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46041bb901664061b44eb699831c4e55" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 36947 } }
I20260925 15:17:16.676505 28887 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46041bb901664061b44eb699831c4e55 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:17:16.678751 28892 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46041bb901664061b44eb699831c4e55 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "46041bb901664061b44eb699831c4e55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46041bb901664061b44eb699831c4e55" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 36947 } } }
I20260925 15:17:16.679608 28892 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46041bb901664061b44eb699831c4e55 [sys.catalog]: This master's current role is: LEADER
I20260925 15:17:16.679170 28893 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46041bb901664061b44eb699831c4e55 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46041bb901664061b44eb699831c4e55. Latest consensus state: current_term: 3 leader_uuid: "46041bb901664061b44eb699831c4e55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46041bb901664061b44eb699831c4e55" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 36947 } } }
I20260925 15:17:16.680320 28893 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46041bb901664061b44eb699831c4e55 [sys.catalog]: This master's current role is: LEADER
I20260925 15:17:16.683549 28898 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:17:16.692245 28898 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:17:16.693806 28898 catalog_manager.cc:1259] Loaded cluster ID: 1a3299685a8e4f32a0950e03622ecab5
I20260925 15:17:16.693910 28898 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:17:16.698508 28898 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:17:16.700153 28898 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 46041bb901664061b44eb699831c4e55: Loaded TSK: 0
I20260925 15:17:16.700302 28898 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 46041bb901664061b44eb699831c4e55: Loaded TSK: 1
F20260925 15:17:16.701428 28898 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fdef81ac9ad google::LogMessage::Fail() at ??:0
@ 0x7fdef81af88a google::LogMessage::SendToLog() at ??:0
@ 0x7fdef81ac4de google::LogMessage::Flush() at ??:0
@ 0x7fdef81ada19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fdefbf107d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fdefbeff73e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fdf0dbdb1c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fdf0dbda783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fdf0dc662a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7fdf0dc660c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7fdf0dcb05e9 std::function<>::operator()() at ??:0
@ 0x7fdf0dbdecdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fdf0dbdcc36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fdf0dc64610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fdf0dc64439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fdefadf8345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fdefadfb91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fdefadfb799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fdefadd6eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fdefe0b06db start_thread at ??:0
@ 0x7fdef704661f clone at ??:0
W20260925 15:17:17.043789 28889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:17:17.044050 28889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:17:17.044153 28889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:17:17.051414 28889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:17:17.051512 28889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:17:17.051646 28889 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.1
I20260925 15:17:17.061012 28889 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:38153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.1.keytab
--principal=kudu/127.12.8.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.1:43327
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.8.1
--webserver_port=45247
--tserver_master_addrs=127.12.8.62:36947
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.1
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:17:17.062160 28889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:17:17.063483 28889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:17:17.076413 28934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:17:17.076457 28935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:17:17.076490 28937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:17:17.077216 28889 server_base.cc:1034] running on GCE node
I20260925 15:17:17.077837 28889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:17:17.078809 28889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:17:17.080014 28889 hybrid_clock.cc:648] HybridClock initialized: now 1790349437079973 us; error 35 us; skew 500 ppm
I20260925 15:17:17.080338 28889 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:17:17 dist-test-slave-p468 krb5kdc[27741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349437, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:17:17.083583 28889 init.cc:377] Logged in from keytab as kudu/127.12.8.1@KRBTEST.COM (short username kudu)
I20260925 15:17:17.085042 28889 webserver.cc:460] Webserver started at http://127.12.8.1:45247/ using document root <none> and password file <none>
I20260925 15:17:17.085737 28889 fs_manager.cc:359] Metadata directory not provided
I20260925 15:17:17.085889 28889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:17:17.091158 28889 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 15:17:17.093686 28944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:17:17.095033 28889 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 15:17:17.095196 28889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-0/wal
uuid: "232b8a44ea8e4bd39da1eed7fcb322e4"
format_stamp: "Formatted at 2026-09-25 15:16:31 on dist-test-slave-p468"
server_key: "3c2680983affd0ece3432972c9ed0d94"
server_key_iv: "7968c95a7ada86ad22f2d74947990827"
server_key_version: "encryptionkey@0"
I20260925 15:17:17.096139 28889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:17:17.108731 28889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:17:17.112360 28889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:17:17.112622 28889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:17:17.113528 28889 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:17:17.114853 28889 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:17:17.114969 28889 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:17:17.115094 28889 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:17:17.115195 28889 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:17:17.148566 28889 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.1:43327
I20260925 15:17:17.148653 29056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.1:43327 every 8 connection(s)
I20260925 15:17:17.149991 28889 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-0/data/info.pb
I20260925 15:17:17.153900 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 28889
I20260925 15:17:17.154695 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.2:45177
--local_ip_for_outbound_sockets=127.12.8.2
--tserver_master_addrs=127.12.8.62:36947
--webserver_port=36279
--webserver_interface=127.12.8.2
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.2.keytab
--principal=kudu/127.12.8.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:38153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:17:17.155159 29059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:17.152832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:36947 (local address 127.12.8.1:54905)
0925 15:17:17.153681 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:17.153931 (+ 250us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:36947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":278,"threads_started":1}
W20260925 15:17:17.155936 29057 heartbeater.cc:644] Failed to heartbeat to 127.12.8.62:36947 (0 consecutive failures): Network error: Failed to ping master at 127.12.8.62:36947: Client connection negotiation failed: client connection to 127.12.8.62:36947: connect: Connection refused (error 111)
I20260925 15:17:17.156870 29059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:17.156450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:36947 (local address 127.12.8.1:52499)
0925 15:17:17.156636 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:17.156725 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:36947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260925 15:17:17.157895 29059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:17.157463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:36947 (local address 127.12.8.1:57565)
0925 15:17:17.157647 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:17.157744 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:36947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 15:17:17.158181 29057 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 15:17:17.475214 29060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:17:17.475474 29060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:17:17.475662 29060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:17:17.482905 29060 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:17:17.483057 29060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:17:17.483273 29060 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.2
I20260925 15:17:17.492930 29060 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:38153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.2.keytab
--principal=kudu/127.12.8.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.2:45177
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.8.2
--webserver_port=36279
--tserver_master_addrs=127.12.8.62:36947
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.2
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:17:17.494252 29060 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:17:17.495553 29060 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:17:17.508788 29065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:17:17.508733 29068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:17:17.508836 29066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:17:17.509546 29060 server_base.cc:1034] running on GCE node
I20260925 15:17:17.510110 29060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:17:17.511031 29060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:17:17.512255 29060 hybrid_clock.cc:648] HybridClock initialized: now 1790349437512205 us; error 47 us; skew 500 ppm
I20260925 15:17:17.512557 29060 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:17:17 dist-test-slave-p468 krb5kdc[27741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349437, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:17:17.515549 29060 init.cc:377] Logged in from keytab as kudu/127.12.8.2@KRBTEST.COM (short username kudu)
I20260925 15:17:17.516974 29060 webserver.cc:460] Webserver started at http://127.12.8.2:36279/ using document root <none> and password file <none>
I20260925 15:17:17.517632 29060 fs_manager.cc:359] Metadata directory not provided
I20260925 15:17:17.517778 29060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:17:17.522842 29060 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.002s
I20260925 15:17:17.525409 29075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:17:17.526697 29060 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 15:17:17.526859 29060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-1/wal
uuid: "7bba3fc3f1e84d7c8254a77f05fd82df"
format_stamp: "Formatted at 2026-09-25 15:16:31 on dist-test-slave-p468"
server_key: "841dbc33588ead37eb9b219a185bc313"
server_key_iv: "df420a438caa3a76f6bfff89b1e9ac9a"
server_key_version: "encryptionkey@0"
I20260925 15:17:17.527789 29060 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:17:17.568459 29060 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:17:17.572386 29060 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:17:17.572665 29060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:17:17.573647 29060 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:17:17.574998 29060 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:17:17.575135 29060 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:17:17.575279 29060 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:17:17.575366 29060 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:17:17.609133 29060 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.2:45177
I20260925 15:17:17.609196 29187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.2:45177 every 8 connection(s)
I20260925 15:17:17.610605 29060 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-1/data/info.pb
I20260925 15:17:17.615401 29190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:17.613570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:36947 (local address 127.12.8.2:41143)
0925 15:17:17.614348 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:17.614550 (+ 202us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:36947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":280,"threads_started":1}
W20260925 15:17:17.616052 29188 heartbeater.cc:644] Failed to heartbeat to 127.12.8.62:36947 (0 consecutive failures): Network error: Failed to ping master at 127.12.8.62:36947: Client connection negotiation failed: client connection to 127.12.8.62:36947: connect: Connection refused (error 111)
I20260925 15:17:17.616807 29190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:17.616502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:36947 (local address 127.12.8.2:60297)
0925 15:17:17.616627 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:17.616689 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:36947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260925 15:17:17.617538 29190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:17.617238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:36947 (local address 127.12.8.2:40947)
0925 15:17:17.617352 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:17.617418 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:36947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 15:17:17.617803 29188 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 15:17:17.618301 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 29060
I20260925 15:17:17.618882 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.3:46683
--local_ip_for_outbound_sockets=127.12.8.3
--tserver_master_addrs=127.12.8.62:36947
--webserver_port=34497
--webserver_interface=127.12.8.3
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.3.keytab
--principal=kudu/127.12.8.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:38153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:17:17.948062 29191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:17:17.948350 29191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:17:17.948443 29191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:17:17.955598 29191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:17:17.955716 29191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:17:17.955829 29191 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.3
I20260925 15:17:17.965390 29191 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:38153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.3.keytab
--principal=kudu/127.12.8.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.3:46683
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.8.3
--webserver_port=34497
--tserver_master_addrs=127.12.8.62:36947
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.3
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:17:17.966624 29191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:17:17.967861 29191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:17:17.981053 29197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:17:17.981115 29199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:17:17.981009 29196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:17:17.981930 29191 server_base.cc:1034] running on GCE node
I20260925 15:17:17.982533 29191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:17:17.983393 29191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:17:17.984647 29191 hybrid_clock.cc:648] HybridClock initialized: now 1790349437984596 us; error 56 us; skew 500 ppm
I20260925 15:17:17.985041 29191 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:17:17 dist-test-slave-p468 krb5kdc[27741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349437, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:17:17.988186 29191 init.cc:377] Logged in from keytab as kudu/127.12.8.3@KRBTEST.COM (short username kudu)
I20260925 15:17:17.989671 29191 webserver.cc:460] Webserver started at http://127.12.8.3:34497/ using document root <none> and password file <none>
I20260925 15:17:17.990448 29191 fs_manager.cc:359] Metadata directory not provided
I20260925 15:17:17.990597 29191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:17:17.995899 29191 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 15:17:17.998677 29206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:17:18.000077 29191 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 15:17:18.000216 29191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-2/wal
uuid: "a2cf2ac0a7a14b00986fab5569735fb8"
format_stamp: "Formatted at 2026-09-25 15:16:32 on dist-test-slave-p468"
server_key: "49a90e388af98b837903ea5dd77191a9"
server_key_iv: "3f16fb4f69de85b6124e5ac0e382c389"
server_key_version: "encryptionkey@0"
I20260925 15:17:18.001272 29191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:17:18.017381 29191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:17:18.021329 29191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:17:18.021552 29191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:17:18.022513 29191 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:17:18.023886 29191 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:17:18.023972 29191 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:17:18.024042 29191 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:17:18.024101 29191 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:17:18.058030 29191 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.3:46683
I20260925 15:17:18.058115 29318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.3:46683 every 8 connection(s)
I20260925 15:17:18.059484 29191 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349162370577-12320-0/minicluster-data/ts-2/data/info.pb
I20260925 15:17:18.062321 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 29191
I20260925 15:17:18.064918 29321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:18.062795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:36947 (local address 127.12.8.3:45399)
0925 15:17:18.063661 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:18.063897 (+ 236us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:36947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":359,"threads_started":1}
W20260925 15:17:18.065563 29319 heartbeater.cc:644] Failed to heartbeat to 127.12.8.62:36947 (0 consecutive failures): Network error: Failed to ping master at 127.12.8.62:36947: Client connection negotiation failed: client connection to 127.12.8.62:36947: connect: Connection refused (error 111)
I20260925 15:17:18.066413 29321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:18.066042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:36947 (local address 127.12.8.3:54827)
0925 15:17:18.066234 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:18.066314 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:36947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260925 15:17:18.067225 29321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:18.066904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:36947 (local address 127.12.8.3:60959)
0925 15:17:18.067046 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:18.067125 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:36947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260925 15:17:18.067459 29319 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 15:17:18.159471 29323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:18.158728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:36947 (local address 127.12.8.1:48741)
0925 15:17:18.159159 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:18.159244 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:36947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":216,"threads_started":1}
I20260925 15:17:18.619026 29324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:18.618294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:36947 (local address 127.12.8.2:54449)
0925 15:17:18.618726 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:18.618808 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:36947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":207,"threads_started":1}
I20260925 15:17:19.068715 29325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:19.067974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:36947 (local address 127.12.8.3:43409)
0925 15:17:19.068406 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:19.068489 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:36947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":205,"threads_started":1}
I20260925 15:17:19.161029 29326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:19.160316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:36947 (local address 127.12.8.1:46351)
0925 15:17:19.160779 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:19.160866 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:36947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":223,"threads_started":1}
I20260925 15:17:19.620635 29327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:19.619909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:36947 (local address 127.12.8.2:51843)
0925 15:17:19.620366 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:19.620453 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:36947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":244,"threads_started":1}
I20260925 15:17:20.070310 29328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:20.069572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:36947 (local address 127.12.8.3:50187)
0925 15:17:20.070011 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:20.070098 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:36947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":219,"threads_started":1}
I20260925 15:17:20.162628 29329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:20.161909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:36947 (local address 127.12.8.1:46143)
0925 15:17:20.162374 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:20.162453 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:36947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":214,"threads_started":1}
I20260925 15:17:20.622550 29330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:20.621705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:36947 (local address 127.12.8.2:58813)
0925 15:17:20.622298 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:20.622379 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:36947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":338,"threads_started":1}
I20260925 15:17:21.071756 29331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:21.071111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:36947 (local address 127.12.8.3:33899)
0925 15:17:21.071520 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:21.071595 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:36947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":213,"threads_started":1}
I20260925 15:17:21.164160 29332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:21.163449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:36947 (local address 127.12.8.1:35131)
0925 15:17:21.163889 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:21.163966 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:36947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":225,"threads_started":1}
I20260925 15:17:21.624375 29333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:21.623606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:36947 (local address 127.12.8.2:47613)
0925 15:17:21.624059 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:21.624136 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:36947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":239,"threads_started":1}
I20260925 15:17:22.073258 29334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:22.072597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:36947 (local address 127.12.8.3:56999)
0925 15:17:22.072999 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:22.073077 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:36947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":205,"threads_started":1}
I20260925 15:17:22.165751 29335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:22.165053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:36947 (local address 127.12.8.1:48751)
0925 15:17:22.165503 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:22.165578 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:36947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":247,"threads_started":1}
I20260925 15:17:22.626075 29336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:22.625282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:36947 (local address 127.12.8.2:50023)
0925 15:17:22.625760 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:22.625862 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:36947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":228,"threads_started":1}
I20260925 15:17:23.075002 29337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:23.074166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:36947 (local address 127.12.8.3:60705)
0925 15:17:23.074728 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:23.074820 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:36947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":258,"threads_started":1}
I20260925 15:17:23.167685 29338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:23.166829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:36947 (local address 127.12.8.1:47273)
0925 15:17:23.167428 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:23.167517 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:36947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":310,"threads_started":1}
I20260925 15:17:23.627813 29339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:23.627021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:36947 (local address 127.12.8.2:52675)
0925 15:17:23.627539 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:23.627641 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:36947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":312,"threads_started":1}
I20260925 15:17:24.076501 29340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:24.075796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:36947 (local address 127.12.8.3:42083)
0925 15:17:24.076246 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:24.076332 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:36947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":239,"threads_started":1}
I20260925 15:17:24.169301 29341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:24.168534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:36947 (local address 127.12.8.1:47859)
0925 15:17:24.168998 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:24.169076 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:36947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":209,"threads_started":1}
I20260925 15:17:24.629604 29342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:24.628775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:36947 (local address 127.12.8.2:53373)
0925 15:17:24.629289 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:24.629386 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:36947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":227,"threads_started":1}
I20260925 15:17:25.078070 29343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:25.077298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:36947 (local address 127.12.8.3:47913)
0925 15:17:25.077740 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:25.077828 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:36947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":192,"threads_started":1}
I20260925 15:17:25.171087 29344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:25.170341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:36947 (local address 127.12.8.1:54685)
0925 15:17:25.170793 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:25.170913 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:36947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":222,"threads_started":1}
I20260925 15:17:25.631232 29345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:25.630547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:36947 (local address 127.12.8.2:42499)
0925 15:17:25.630990 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:25.631071 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:36947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":240,"threads_started":1}
I20260925 15:17:26.079834 29346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:26.078927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:36947 (local address 127.12.8.3:41579)
0925 15:17:26.079522 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:26.079602 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:36947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":316,"threads_started":1}
I20260925 15:17:26.172621 29347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:26.171945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:36947 (local address 127.12.8.1:37637)
0925 15:17:26.172371 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:26.172450 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:36947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":212,"threads_started":1}
I20260925 15:17:26.632764 29348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:26.632050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:36947 (local address 127.12.8.2:58347)
0925 15:17:26.632508 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:26.632591 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:36947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":249,"threads_started":1}
I20260925 15:17:27.081434 29349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:27.080623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:36947 (local address 127.12.8.3:51217)
0925 15:17:27.081138 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:27.081225 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:36947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":257,"threads_started":1}
I20260925 15:17:27.174139 29350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:27.173490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:36947 (local address 127.12.8.1:51163)
0925 15:17:27.173895 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:27.173974 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:36947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":225,"threads_started":1}
I20260925 15:17:27.634377 29351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:27.633619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:36947 (local address 127.12.8.2:46601)
0925 15:17:27.634058 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:27.634139 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:36947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":242,"threads_started":1}
I20260925 15:17:28.083138 29352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:28.082436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:36947 (local address 127.12.8.3:39939)
0925 15:17:28.082887 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:28.082974 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:36947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":239,"threads_started":1}
I20260925 15:17:28.175756 29353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:28.174991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:36947 (local address 127.12.8.1:55665)
0925 15:17:28.175447 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:28.175536 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:36947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":235,"threads_started":1}
I20260925 15:17:28.635919 29354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:28.635198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:36947 (local address 127.12.8.2:46533)
0925 15:17:28.635635 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:28.635722 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:36947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":213,"threads_started":1}
I20260925 15:17:29.084749 29355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:29.084058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:36947 (local address 127.12.8.3:46069)
0925 15:17:29.084494 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:29.084578 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:36947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":222,"threads_started":1}
I20260925 15:17:29.177379 29356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:29.176658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:36947 (local address 127.12.8.1:35041)
0925 15:17:29.177127 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:29.177212 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:36947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":237,"threads_started":1}
I20260925 15:17:29.637457 29357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:29.636804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:36947 (local address 127.12.8.2:46943)
0925 15:17:29.637218 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:29.637297 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:36947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":208,"threads_started":1}
I20260925 15:17:30.086457 29358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:30.085680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:36947 (local address 127.12.8.3:54033)
0925 15:17:30.086135 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:30.086293 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:36947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":224,"threads_started":1}
I20260925 15:17:30.179314 29359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:30.178497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:36947 (local address 127.12.8.1:48541)
0925 15:17:30.179020 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:30.179114 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:36947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":247,"threads_started":1}
I20260925 15:17:30.639098 29360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:30.638389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:36947 (local address 127.12.8.2:59733)
0925 15:17:30.638818 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:30.638898 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:36947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":218,"threads_started":1}
I20260925 15:17:31.088029 29361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:31.087310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:36947 (local address 127.12.8.3:46819)
0925 15:17:31.087764 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:31.087844 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:36947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":218,"threads_started":1}
I20260925 15:17:31.180872 29362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:31.180151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:36947 (local address 127.12.8.1:53911)
0925 15:17:31.180616 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:31.180696 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:36947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":220,"threads_started":1}
I20260925 15:17:31.640638 29363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:31.639959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:36947 (local address 127.12.8.2:40437)
0925 15:17:31.640383 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:31.640460 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:36947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":227,"threads_started":1}
I20260925 15:17:32.089906 29364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:32.088961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:36947 (local address 127.12.8.3:40163)
0925 15:17:32.089544 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:32.089671 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:36947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":279,"threads_started":1}
I20260925 15:17:32.182631 29365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:32.181905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:36947 (local address 127.12.8.1:54539)
0925 15:17:32.182369 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:32.182461 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:36947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":216,"threads_started":1}
I20260925 15:17:32.642505 29366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:32.641721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:36947 (local address 127.12.8.2:46235)
0925 15:17:32.642162 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:32.642321 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:36947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":220,"threads_started":1}
I20260925 15:17:33.065083 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 28889
I20260925 15:17:33.091886 29367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:33.066352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47058 (local address 127.12.8.1:43327)
0925 15:17:33.066963 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:33.066976 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:33.067014 (+ 38us) server_negotiation.cc:407] Connection header received
0925 15:17:33.068582 (+ 1568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:33.068590 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:33.069019 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:33.069299 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:17:33.070773 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:33.072048 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:33.073315 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:33.074027 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:33.074171 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:33.085434 (+ 11263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:17:33.085442 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:17:33.085457 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:17:33.085500 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:17:33.088512 (+ 3012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:17:33.089150 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:17:33.089158 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:17:33.089166 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:17:33.089339 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:17:33.089664 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:17:33.089667 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:17:33.089669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:17:33.090248 (+ 579us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:17:33.090564 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:33.091050 (+ 486us) server_negotiation.cc:299] Negotiation successful
0925 15:17:33.091456 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":346,"threads_started":1}
I20260925 15:17:33.092526 29368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:33.091632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:36947 (local address 127.12.8.3:44907)
0925 15:17:33.092184 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:33.092310 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.62:36947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":312,"threads_started":1}
I20260925 15:17:33.093969 29052 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:17:33.177551 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 28889
I20260925 15:17:33.194243 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 29060
I20260925 15:17:33.210736 29370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:33.195115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48750 (local address 127.12.8.2:45177)
0925 15:17:33.195636 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:33.195650 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:33.195687 (+ 37us) server_negotiation.cc:407] Connection header received
0925 15:17:33.197120 (+ 1433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:33.197129 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:33.197542 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:33.197826 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:17:33.199252 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:33.200462 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:33.201602 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:33.202330 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:33.202469 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:33.204988 (+ 2519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:17:33.204996 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:17:33.205009 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:17:33.205043 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:17:33.207941 (+ 2898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:17:33.208552 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:17:33.208558 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:17:33.208563 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:17:33.208724 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:17:33.209030 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:17:33.209033 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:17:33.209035 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:17:33.209517 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:17:33.209749 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:33.210083 (+ 334us) server_negotiation.cc:299] Negotiation successful
0925 15:17:33.210468 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":251,"threads_started":1}
I20260925 15:17:33.212158 29183 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:17:33.294782 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 29060
I20260925 15:17:33.311928 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 29191
I20260925 15:17:33.330448 29372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:33.312977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60990 (local address 127.12.8.3:46683)
0925 15:17:33.313664 (+ 687us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:33.313679 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:33.313726 (+ 47us) server_negotiation.cc:407] Connection header received
0925 15:17:33.315600 (+ 1874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:33.315610 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:33.316084 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:33.316419 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:17:33.317650 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:33.319152 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:33.320419 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:33.321168 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:33.321310 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:33.323976 (+ 2666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:17:33.323984 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:17:33.323999 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:17:33.324038 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:17:33.327218 (+ 3180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:17:33.327904 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:17:33.327911 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:17:33.327917 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:17:33.328076 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:17:33.328472 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:17:33.328479 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:17:33.328481 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:17:33.329027 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:17:33.329257 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:33.329728 (+ 471us) server_negotiation.cc:299] Negotiation successful
0925 15:17:33.330058 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":329,"threads_started":1}
I20260925 15:17:33.331826 29314 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:17:33.415284 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 29191
2026-09-25T15:17:33Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64442 ms)
[----------] 30 tests from SecurityITest (291011 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349162370577-12320-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:17:33 dist-test-slave-p468 krb5kdc[29376](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:17:33 dist-test-slave-p468 krb5kdc[29376](info): set up 2 sockets
Sep 25 15:17:33 dist-test-slave-p468 krb5kdc[29376](info): commencing operation
krb5kdc: starting...
W20260925 15:17:34.534371 12320 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349162370577-12320-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349162370577-12320-0/krb5kdc/test-user.keytab.
Sep 25 15:17:34 dist-test-slave-p468 krb5kdc[29376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:17:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:17:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.8.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab.
Entry for principal kudu/127.12.8.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab.
I20260925 15:17:34.771919 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349162370577-12320-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349162370577-12320-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349162370577-12320-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.62:38713
--webserver_interface=127.12.8.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab
--principal=kudu/127.12.8.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:38641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.62:38713
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:17:35.095850 29389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:17:35.096182 29389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:17:35.096362 29389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:17:35.104804 29389 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:17:35.104947 29389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:17:35.105052 29389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:17:35.105165 29389 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:17:35.105252 29389 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:17:35.115381 29389 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:38641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349162370577-12320-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349162370577-12320-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.8.62:38713
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab
--principal=kudu/127.12.8.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.62:38713
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.8.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349162370577-12320-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:17:35.116706 29389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:17:35.117985 29389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:17:35.129484 29395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:17:35.129542 29397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:17:35.129455 29394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:17:35.130275 29389 server_base.cc:1034] running on GCE node
I20260925 15:17:35.131060 29389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:17:35.132467 29389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:17:35.133700 29389 hybrid_clock.cc:648] HybridClock initialized: now 1790349455133663 us; error 47 us; skew 500 ppm
I20260925 15:17:35.134146 29389 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:17:35 dist-test-slave-p468 krb5kdc[29376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349455, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:17:35.137120 29389 init.cc:377] Logged in from keytab as kudu/127.12.8.62@KRBTEST.COM (short username kudu)
I20260925 15:17:35.138645 29389 webserver.cc:460] Webserver started at http://127.12.8.62:35195/ using document root <none> and password file <none>
I20260925 15:17:35.139321 29389 fs_manager.cc:359] Metadata directory not provided
I20260925 15:17:35.139465 29389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:17:35.139766 29389 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:17:35.142709 29389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349162370577-12320-0/minicluster-data/master-0/data/instance:
uuid: "4addb03da683435f81e6cec78ccf2c15"
format_stamp: "Formatted at 2026-09-25 15:17:35 on dist-test-slave-p468"
server_key: "b43ea41c97e9d72ab7f6cfc5efae3e8a"
server_key_iv: "7f40fb9da356b353984fabf6dca5b21f"
server_key_version: "encryptionkey@0"
I20260925 15:17:35.143478 29389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349162370577-12320-0/minicluster-data/master-0/wal/instance:
uuid: "4addb03da683435f81e6cec78ccf2c15"
format_stamp: "Formatted at 2026-09-25 15:17:35 on dist-test-slave-p468"
server_key: "b43ea41c97e9d72ab7f6cfc5efae3e8a"
server_key_iv: "7f40fb9da356b353984fabf6dca5b21f"
server_key_version: "encryptionkey@0"
I20260925 15:17:35.148514 29389 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 15:17:35.151873 29404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:17:35.153208 29389 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 15:17:35.153414 29389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349162370577-12320-0/minicluster-data/master-0/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349162370577-12320-0/minicluster-data/master-0/wal
uuid: "4addb03da683435f81e6cec78ccf2c15"
format_stamp: "Formatted at 2026-09-25 15:17:35 on dist-test-slave-p468"
server_key: "b43ea41c97e9d72ab7f6cfc5efae3e8a"
server_key_iv: "7f40fb9da356b353984fabf6dca5b21f"
server_key_version: "encryptionkey@0"
I20260925 15:17:35.153612 29389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349162370577-12320-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349162370577-12320-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349162370577-12320-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:17:35.167848 29389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:17:35.172084 29389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:17:35.172389 29389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:17:35.191069 29389 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.62:38713
I20260925 15:17:35.191104 29455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.62:38713 every 8 connection(s)
I20260925 15:17:35.192447 29389 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
I20260925 15:17:35.194408 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 29389
I20260925 15:17:35.194628 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349162370577-12320-0/minicluster-data/master-0/wal/instance
I20260925 15:17:35.195469 12320 external_mini_cluster.cc:1428] Setting key 9e148e36bdc3fd009ddce5efc58414a0
I20260925 15:17:35.196915 29456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:17:35.208447 29456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4addb03da683435f81e6cec78ccf2c15: Bootstrap starting.
Sep 25 15:17:35 dist-test-slave-p468 krb5kdc[29376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
I20260925 15:17:35.212244 29456 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4addb03da683435f81e6cec78ccf2c15: Neither blocks nor log segments found. Creating new log.
I20260925 15:17:35.213292 29456 log.cc:824] T 00000000000000000000000000000000 P 4addb03da683435f81e6cec78ccf2c15: Log is configured to *not* fsync() on all Append() calls
I20260925 15:17:35.216552 29456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4addb03da683435f81e6cec78ccf2c15: No bootstrap required, opened a new log
I20260925 15:17:35.217505 29459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:35.197494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55224 (local address 127.12.8.62:38713)
0925 15:17:35.198325 (+ 831us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:35.198340 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:35.198388 (+ 48us) server_negotiation.cc:407] Connection header received
0925 15:17:35.200010 (+ 1622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:35.200030 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:35.200391 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:35.200787 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:17:35.202250 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:35.203543 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:35.204917 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:35.205585 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:35.205798 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:35.210383 (+ 4585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:17:35.210391 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:17:35.210404 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:17:35.210441 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:17:35.213536 (+ 3095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:17:35.214356 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:17:35.214365 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:17:35.214376 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:17:35.214576 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:17:35.215034 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:17:35.215037 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:17:35.215039 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:17:35.215517 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:17:35.215718 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:35.216218 (+ 500us) server_negotiation.cc:299] Negotiation successful
0925 15:17:35.216511 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":332,"threads_started":1}
I20260925 15:17:35.223971 29456 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4addb03da683435f81e6cec78ccf2c15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4addb03da683435f81e6cec78ccf2c15" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 38713 } }
I20260925 15:17:35.224344 29456 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4addb03da683435f81e6cec78ccf2c15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:17:35.224465 29456 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4addb03da683435f81e6cec78ccf2c15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4addb03da683435f81e6cec78ccf2c15, State: Initialized, Role: FOLLOWER
I20260925 15:17:35.224987 29456 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4addb03da683435f81e6cec78ccf2c15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4addb03da683435f81e6cec78ccf2c15" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 38713 } }
I20260925 15:17:35.225238 29456 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4addb03da683435f81e6cec78ccf2c15 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:17:35.225389 29456 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4addb03da683435f81e6cec78ccf2c15 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:17:35.225566 29456 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4addb03da683435f81e6cec78ccf2c15 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:17:35.227805 29456 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4addb03da683435f81e6cec78ccf2c15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4addb03da683435f81e6cec78ccf2c15" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 38713 } }
I20260925 15:17:35.228297 29456 leader_election.cc:302] T 00000000000000000000000000000000 P 4addb03da683435f81e6cec78ccf2c15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4addb03da683435f81e6cec78ccf2c15; no voters:
I20260925 15:17:35.228936 29456 leader_election.cc:288] T 00000000000000000000000000000000 P 4addb03da683435f81e6cec78ccf2c15 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:17:35.229070 29461 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4addb03da683435f81e6cec78ccf2c15 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:17:35.230036 29461 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4addb03da683435f81e6cec78ccf2c15 [term 1 LEADER]: Becoming Leader. State: Replica: 4addb03da683435f81e6cec78ccf2c15, State: Running, Role: LEADER
I20260925 15:17:35.230615 29461 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4addb03da683435f81e6cec78ccf2c15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4addb03da683435f81e6cec78ccf2c15" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 38713 } }
I20260925 15:17:35.231254 29456 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4addb03da683435f81e6cec78ccf2c15 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:17:35.233845 29462 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4addb03da683435f81e6cec78ccf2c15 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4addb03da683435f81e6cec78ccf2c15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4addb03da683435f81e6cec78ccf2c15" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 38713 } } }
I20260925 15:17:35.233871 29463 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4addb03da683435f81e6cec78ccf2c15 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4addb03da683435f81e6cec78ccf2c15. Latest consensus state: current_term: 1 leader_uuid: "4addb03da683435f81e6cec78ccf2c15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4addb03da683435f81e6cec78ccf2c15" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 38713 } } }
I20260925 15:17:35.234402 29463 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4addb03da683435f81e6cec78ccf2c15 [sys.catalog]: This master's current role is: LEADER
I20260925 15:17:35.235049 29462 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4addb03da683435f81e6cec78ccf2c15 [sys.catalog]: This master's current role is: LEADER
I20260925 15:17:35.235314 29467 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:17:35.241251 29467 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:17:35.252331 29467 catalog_manager.cc:1347] Generated new cluster ID: 14d5501ccbb74dd595c65bc4ff795f97
I20260925 15:17:35.252496 29467 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:17:35.265679 29467 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:17:35.267604 29467 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:17:35.279304 29467 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4addb03da683435f81e6cec78ccf2c15: Generated new TSK 0
I20260925 15:17:35.280380 29467 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.8.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.1.keytab.
Entry for principal kudu/127.12.8.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.1.keytab.
I20260925 15:17:35.362309 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349162370577-12320-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349162370577-12320-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349162370577-12320-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349162370577-12320-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.1:0
--local_ip_for_outbound_sockets=127.12.8.1
--webserver_interface=127.12.8.1
--webserver_port=0
--tserver_master_addrs=127.12.8.62:38713
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.1.keytab
--principal=kudu/127.12.8.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:38641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:17:35.688692 29482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:17:35.689029 29482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:17:35.689196 29482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:17:35.696491 29482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:17:35.696645 29482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:17:35.696820 29482 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.1
I20260925 15:17:35.706552 29482 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:38641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349162370577-12320-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349162370577-12320-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.1.keytab
--principal=kudu/127.12.8.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349162370577-12320-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.8.1
--webserver_port=0
--tserver_master_addrs=127.12.8.62:38713
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.1
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349162370577-12320-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:17:35.707901 29482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:17:35.709141 29482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:17:35.721639 29487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:17:35.721608 29490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:17:35.721680 29488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:17:35.722857 29482 server_base.cc:1034] running on GCE node
I20260925 15:17:35.723431 29482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:17:35.724314 29482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:17:35.725557 29482 hybrid_clock.cc:648] HybridClock initialized: now 1790349455725503 us; error 58 us; skew 500 ppm
I20260925 15:17:35.725869 29482 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:17:35 dist-test-slave-p468 krb5kdc[29376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349455, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:17:35.728957 29482 init.cc:377] Logged in from keytab as kudu/127.12.8.1@KRBTEST.COM (short username kudu)
I20260925 15:17:35.730472 29482 webserver.cc:460] Webserver started at http://127.12.8.1:40043/ using document root <none> and password file <none>
I20260925 15:17:35.731284 29482 fs_manager.cc:359] Metadata directory not provided
I20260925 15:17:35.731459 29482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:17:35.731807 29482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:17:35.734822 29482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349162370577-12320-0/minicluster-data/ts-0/data/instance:
uuid: "557bedf5a03c4cd696ae9129d1ef4339"
format_stamp: "Formatted at 2026-09-25 15:17:35 on dist-test-slave-p468"
server_key: "b5210bc19487f8de71802da83bf1ce03"
server_key_iv: "45a4399fe0f1ccecbd69e709fdafdedc"
server_key_version: "encryptionkey@0"
I20260925 15:17:35.735546 29482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349162370577-12320-0/minicluster-data/ts-0/wal/instance:
uuid: "557bedf5a03c4cd696ae9129d1ef4339"
format_stamp: "Formatted at 2026-09-25 15:17:35 on dist-test-slave-p468"
server_key: "b5210bc19487f8de71802da83bf1ce03"
server_key_iv: "45a4399fe0f1ccecbd69e709fdafdedc"
server_key_version: "encryptionkey@0"
I20260925 15:17:35.740715 29482 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 15:17:35.744023 29497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:17:35.745287 29482 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 15:17:35.745450 29482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349162370577-12320-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349162370577-12320-0/minicluster-data/ts-0/wal
uuid: "557bedf5a03c4cd696ae9129d1ef4339"
format_stamp: "Formatted at 2026-09-25 15:17:35 on dist-test-slave-p468"
server_key: "b5210bc19487f8de71802da83bf1ce03"
server_key_iv: "45a4399fe0f1ccecbd69e709fdafdedc"
server_key_version: "encryptionkey@0"
I20260925 15:17:35.745642 29482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349162370577-12320-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349162370577-12320-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349162370577-12320-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:17:35.760342 29482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:17:35.764222 29482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:17:35.764515 29482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:17:35.765512 29482 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:17:35.766855 29482 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:17:35.766928 29482 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:17:35.767088 29482 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:17:35.767148 29482 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:17:35.801106 29482 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.1:43981
I20260925 15:17:35.801190 29609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.1:43981 every 8 connection(s)
I20260925 15:17:35.802452 29482 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349162370577-12320-0/minicluster-data/ts-0/data/info.pb
I20260925 15:17:35.806474 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 29482
I20260925 15:17:35.806696 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349162370577-12320-0/minicluster-data/ts-0/wal/instance
I20260925 15:17:35.807720 12320 external_mini_cluster.cc:1428] Setting key 9f0b21ebbeadd2f45baa078211dbe429
Sep 25 15:17:35 dist-test-slave-p468 krb5kdc[29376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349455, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.1@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
I20260925 15:17:35.828508 29612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:35.805205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.1:58171 (local address 127.12.8.62:38713)
0925 15:17:35.805676 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:35.805680 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:35.806720 (+ 1040us) server_negotiation.cc:407] Connection header received
0925 15:17:35.810241 (+ 3521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:35.810248 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:35.810336 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:35.810438 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:17:35.812640 (+ 2202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:35.813413 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:35.815963 (+ 2550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:35.816395 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:35.816493 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:35.821045 (+ 4552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:17:35.821054 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:17:35.821060 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:17:35.821105 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:17:35.824938 (+ 3833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:17:35.825729 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:17:35.825738 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:17:35.825745 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:17:35.825905 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:17:35.826384 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:17:35.826392 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:17:35.826399 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:17:35.826816 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:17:35.826989 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:35.827678 (+ 689us) server_negotiation.cc:299] Negotiation successful
0925 15:17:35.827974 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":250,"threads_started":1}
I20260925 15:17:35.829092 29613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:35.805642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:38713 (local address 127.12.8.1:58171)
0925 15:17:35.806520 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:35.806576 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 15:17:35.809861 (+ 3285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:17:35.810760 (+ 899us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:17:35.810774 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:17:35.811582 (+ 808us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:17:35.812445 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:35.812473 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:35.813589 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:35.813597 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:35.815799 (+ 2202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:35.815818 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:35.816628 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:35.816637 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:35.816977 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:35.817725 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:17:35.817768 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:17:35.820799 (+ 3031us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:17:35.825160 (+ 4361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:17:35.825167 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:17:35.825179 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:17:35.825583 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:17:35.826017 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:17:35.826020 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:17:35.826022 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:17:35.826251 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:17:35.826946 (+ 695us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:17:35.826951 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:17:35.827257 (+ 306us) client_negotiation.cc:769] Sending connection context
0925 15:17:35.827479 (+ 222us) client_negotiation.cc:240] Negotiation successful
0925 15:17:35.827868 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":319,"threads_started":1}
I20260925 15:17:35.830790 29610 heartbeater.cc:347] Connected to a master server at 127.12.8.62:38713
I20260925 15:17:35.831097 29610 heartbeater.cc:464] Registering TS with master...
I20260925 15:17:35.831868 29610 heartbeater.cc:511] Master 127.12.8.62:38713 requested a full tablet report, sending...
I20260925 15:17:35.834455 29421 ts_manager.cc:194] Registered new tserver with Master: 557bedf5a03c4cd696ae9129d1ef4339 (127.12.8.1:43981)
I20260925 15:17:35.836864 29421 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.1@KRBTEST.COM'} at 127.12.8.1:58171
WARNING: no policy specified for kudu/127.12.8.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.2.keytab.
Entry for principal kudu/127.12.8.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.2.keytab.
I20260925 15:17:35.870141 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349162370577-12320-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349162370577-12320-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349162370577-12320-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349162370577-12320-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.2:0
--local_ip_for_outbound_sockets=127.12.8.2
--webserver_interface=127.12.8.2
--webserver_port=0
--tserver_master_addrs=127.12.8.62:38713
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.2.keytab
--principal=kudu/127.12.8.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:38641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:17:36.203238 29616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:17:36.203488 29616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:17:36.203668 29616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:17:36.210886 29616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:17:36.210987 29616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:17:36.211210 29616 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.2
I20260925 15:17:36.220911 29616 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:38641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349162370577-12320-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349162370577-12320-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.2.keytab
--principal=kudu/127.12.8.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349162370577-12320-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.8.2
--webserver_port=0
--tserver_master_addrs=127.12.8.62:38713
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.2
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349162370577-12320-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:17:36.222236 29616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:17:36.223532 29616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:17:36.236420 29621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:17:36.236548 29622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:17:36.236517 29624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:17:36.237439 29616 server_base.cc:1034] running on GCE node
I20260925 15:17:36.238044 29616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:17:36.239007 29616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:17:36.240199 29616 hybrid_clock.cc:648] HybridClock initialized: now 1790349456240134 us; error 67 us; skew 500 ppm
I20260925 15:17:36.240518 29616 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:17:36 dist-test-slave-p468 krb5kdc[29376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349456, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:17:36.243990 29616 init.cc:377] Logged in from keytab as kudu/127.12.8.2@KRBTEST.COM (short username kudu)
I20260925 15:17:36.245518 29616 webserver.cc:460] Webserver started at http://127.12.8.2:45607/ using document root <none> and password file <none>
I20260925 15:17:36.246291 29616 fs_manager.cc:359] Metadata directory not provided
I20260925 15:17:36.246433 29616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:17:36.246737 29616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:17:36.249749 29616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349162370577-12320-0/minicluster-data/ts-1/data/instance:
uuid: "3f98e52717de46769158682003c72b55"
format_stamp: "Formatted at 2026-09-25 15:17:36 on dist-test-slave-p468"
server_key: "5a3bf28f91e0cd78167ac26c2e0a63c9"
server_key_iv: "ae35408ff7acb306689e3498f1805b3f"
server_key_version: "encryptionkey@0"
I20260925 15:17:36.250573 29616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349162370577-12320-0/minicluster-data/ts-1/wal/instance:
uuid: "3f98e52717de46769158682003c72b55"
format_stamp: "Formatted at 2026-09-25 15:17:36 on dist-test-slave-p468"
server_key: "5a3bf28f91e0cd78167ac26c2e0a63c9"
server_key_iv: "ae35408ff7acb306689e3498f1805b3f"
server_key_version: "encryptionkey@0"
I20260925 15:17:36.255584 29616 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 15:17:36.259037 29631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:17:36.260339 29616 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 15:17:36.260500 29616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349162370577-12320-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349162370577-12320-0/minicluster-data/ts-1/wal
uuid: "3f98e52717de46769158682003c72b55"
format_stamp: "Formatted at 2026-09-25 15:17:36 on dist-test-slave-p468"
server_key: "5a3bf28f91e0cd78167ac26c2e0a63c9"
server_key_iv: "ae35408ff7acb306689e3498f1805b3f"
server_key_version: "encryptionkey@0"
I20260925 15:17:36.260694 29616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349162370577-12320-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349162370577-12320-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349162370577-12320-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:17:36.276391 29616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:17:36.280792 29616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:17:36.281096 29616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:17:36.282037 29616 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:17:36.283378 29616 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:17:36.283448 29616 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:17:36.283629 29616 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:17:36.283679 29616 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:17:36.315313 29616 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.2:42297
I20260925 15:17:36.315353 29743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.2:42297 every 8 connection(s)
I20260925 15:17:36.316654 29616 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349162370577-12320-0/minicluster-data/ts-1/data/info.pb
I20260925 15:17:36.323242 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 29616
I20260925 15:17:36.323452 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349162370577-12320-0/minicluster-data/ts-1/wal/instance
I20260925 15:17:36.324052 12320 external_mini_cluster.cc:1428] Setting key 7011d8a5bbcae7523c50e846042049e3
Sep 25 15:17:36 dist-test-slave-p468 krb5kdc[29376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349456, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.2@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
I20260925 15:17:36.339182 29612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:36.319411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.2:49313 (local address 127.12.8.62:38713)
0925 15:17:36.319598 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:36.319603 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:36.320744 (+ 1141us) server_negotiation.cc:407] Connection header received
0925 15:17:36.322550 (+ 1806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:36.322554 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:36.322626 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:36.322718 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:17:36.324818 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:36.325679 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:36.327450 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:36.328222 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:36.328383 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:36.332371 (+ 3988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:17:36.332376 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:17:36.332379 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:17:36.332414 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:17:36.335897 (+ 3483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:17:36.336625 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:17:36.336633 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:17:36.336640 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:17:36.336801 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:17:36.337207 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:17:36.337214 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:17:36.337220 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:17:36.337603 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:17:36.337759 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:36.338482 (+ 723us) server_negotiation.cc:299] Negotiation successful
0925 15:17:36.338744 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 15:17:36.339864 29746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:36.319906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:38713 (local address 127.12.8.2:49313)
0925 15:17:36.320581 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:36.320622 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 15:17:36.322303 (+ 1681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:17:36.322985 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:17:36.322999 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:17:36.323811 (+ 812us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:17:36.324600 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:36.324626 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:36.325895 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:36.325902 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:36.327292 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:36.327306 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:36.328427 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:36.328432 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:36.328628 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:36.329328 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:17:36.329351 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:17:36.332140 (+ 2789us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:17:36.336042 (+ 3902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:17:36.336049 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:17:36.336062 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:17:36.336491 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:17:36.336908 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:17:36.336911 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:17:36.336913 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:17:36.337092 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:17:36.337721 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:17:36.337727 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:17:36.338035 (+ 308us) client_negotiation.cc:769] Sending connection context
0925 15:17:36.338339 (+ 304us) client_negotiation.cc:240] Negotiation successful
0925 15:17:36.338724 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":264,"threads_started":1}
I20260925 15:17:36.341269 29744 heartbeater.cc:347] Connected to a master server at 127.12.8.62:38713
I20260925 15:17:36.341578 29744 heartbeater.cc:464] Registering TS with master...
I20260925 15:17:36.342346 29744 heartbeater.cc:511] Master 127.12.8.62:38713 requested a full tablet report, sending...
I20260925 15:17:36.343729 29421 ts_manager.cc:194] Registered new tserver with Master: 3f98e52717de46769158682003c72b55 (127.12.8.2:42297)
I20260925 15:17:36.344533 29421 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.2@KRBTEST.COM'} at 127.12.8.2:49313
WARNING: no policy specified for kudu/127.12.8.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.3.keytab.
Entry for principal kudu/127.12.8.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.3.keytab.
I20260925 15:17:36.379998 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349162370577-12320-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349162370577-12320-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349162370577-12320-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349162370577-12320-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.3:0
--local_ip_for_outbound_sockets=127.12.8.3
--webserver_interface=127.12.8.3
--webserver_port=0
--tserver_master_addrs=127.12.8.62:38713
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.3.keytab
--principal=kudu/127.12.8.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:38641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:17:36.702678 29749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:17:36.703008 29749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:17:36.703188 29749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:17:36.710322 29749 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:17:36.710482 29749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:17:36.710675 29749 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.3
I20260925 15:17:36.720377 29749 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:38641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349162370577-12320-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349162370577-12320-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.3.keytab
--principal=kudu/127.12.8.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349162370577-12320-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.8.3
--webserver_port=0
--tserver_master_addrs=127.12.8.62:38713
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.3
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349162370577-12320-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:17:36.721745 29749 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:17:36.723111 29749 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:17:36.736886 29754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:17:36.736809 29755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:17:36.736840 29757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:17:36.743155 29749 server_base.cc:1034] running on GCE node
I20260925 15:17:36.743826 29749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:17:36.744844 29749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:17:36.746037 29749 hybrid_clock.cc:648] HybridClock initialized: now 1790349456745968 us; error 62 us; skew 500 ppm
I20260925 15:17:36.746414 29749 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:17:36 dist-test-slave-p468 krb5kdc[29376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349456, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:17:36.749550 29749 init.cc:377] Logged in from keytab as kudu/127.12.8.3@KRBTEST.COM (short username kudu)
I20260925 15:17:36.751157 29749 webserver.cc:460] Webserver started at http://127.12.8.3:39605/ using document root <none> and password file <none>
I20260925 15:17:36.751907 29749 fs_manager.cc:359] Metadata directory not provided
I20260925 15:17:36.752040 29749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:17:36.752359 29749 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:17:36.755364 29749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349162370577-12320-0/minicluster-data/ts-2/data/instance:
uuid: "220dbeb2675142febdc8769bf9f71b97"
format_stamp: "Formatted at 2026-09-25 15:17:36 on dist-test-slave-p468"
server_key: "cc1e775ec5ae287d3998eccb33ffca4b"
server_key_iv: "8fd335a72cc6e92a2320481d12c72a53"
server_key_version: "encryptionkey@0"
I20260925 15:17:36.756135 29749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349162370577-12320-0/minicluster-data/ts-2/wal/instance:
uuid: "220dbeb2675142febdc8769bf9f71b97"
format_stamp: "Formatted at 2026-09-25 15:17:36 on dist-test-slave-p468"
server_key: "cc1e775ec5ae287d3998eccb33ffca4b"
server_key_iv: "8fd335a72cc6e92a2320481d12c72a53"
server_key_version: "encryptionkey@0"
I20260925 15:17:36.761329 29749 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 15:17:36.764705 29764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:17:36.766013 29749 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 15:17:36.766180 29749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349162370577-12320-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349162370577-12320-0/minicluster-data/ts-2/wal
uuid: "220dbeb2675142febdc8769bf9f71b97"
format_stamp: "Formatted at 2026-09-25 15:17:36 on dist-test-slave-p468"
server_key: "cc1e775ec5ae287d3998eccb33ffca4b"
server_key_iv: "8fd335a72cc6e92a2320481d12c72a53"
server_key_version: "encryptionkey@0"
I20260925 15:17:36.766418 29749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349162370577-12320-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349162370577-12320-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349162370577-12320-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:17:36.790063 29749 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:17:36.793823 29749 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:17:36.794049 29749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:17:36.794951 29749 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:17:36.796133 29749 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:17:36.796202 29749 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:17:36.796280 29749 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:17:36.796324 29749 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:17:36.830360 29749 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.3:33727
I20260925 15:17:36.830441 29876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.3:33727 every 8 connection(s)
I20260925 15:17:36.831600 29749 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349162370577-12320-0/minicluster-data/ts-2/data/info.pb
I20260925 15:17:36.832710 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 29749
I20260925 15:17:36.832917 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349162370577-12320-0/minicluster-data/ts-2/wal/instance
I20260925 15:17:36.833492 12320 external_mini_cluster.cc:1428] Setting key e6345d74ef84025713b2c6e119d5e061
I20260925 15:17:36.840051 29610 heartbeater.cc:503] Master 127.12.8.62:38713 was elected leader, sending a full tablet report...
Sep 25 15:17:36 dist-test-slave-p468 krb5kdc[29376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349456, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.3@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
I20260925 15:17:36.852842 29612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:36.834439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.3:58851 (local address 127.12.8.62:38713)
0925 15:17:36.834661 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:36.834668 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:36.835860 (+ 1192us) server_negotiation.cc:407] Connection header received
0925 15:17:36.837916 (+ 2056us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:36.837921 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:36.838009 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:36.838145 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:17:36.839662 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:36.840785 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:36.842503 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:36.842902 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:36.843039 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:36.846507 (+ 3468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:17:36.846520 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:17:36.846523 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:17:36.846554 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:17:36.849821 (+ 3267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:17:36.850589 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:17:36.850594 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:17:36.850598 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:17:36.850721 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:17:36.851147 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:17:36.851152 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:17:36.851157 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:17:36.851438 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:17:36.851600 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:36.852288 (+ 688us) server_negotiation.cc:299] Negotiation successful
0925 15:17:36.852504 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260925 15:17:36.853744 29879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:36.834869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:38713 (local address 127.12.8.3:58851)
0925 15:17:36.835687 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:36.835737 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 15:17:36.837649 (+ 1912us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:17:36.838414 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:17:36.838423 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:17:36.838948 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:17:36.839498 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:36.839516 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:36.840919 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:36.840924 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:36.842356 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:36.842368 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:36.843120 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:36.843129 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:36.843359 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:36.844029 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:17:36.844051 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:17:36.846299 (+ 2248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:17:36.849959 (+ 3660us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:17:36.849968 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:17:36.849984 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:17:36.850463 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:17:36.850838 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:17:36.850843 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:17:36.850847 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:17:36.851026 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:17:36.851579 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:17:36.851586 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:17:36.851904 (+ 318us) client_negotiation.cc:769] Sending connection context
0925 15:17:36.852157 (+ 253us) client_negotiation.cc:240] Negotiation successful
0925 15:17:36.852662 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":304,"threads_started":1}
I20260925 15:17:36.855139 29877 heartbeater.cc:347] Connected to a master server at 127.12.8.62:38713
I20260925 15:17:36.855443 29877 heartbeater.cc:464] Registering TS with master...
I20260925 15:17:36.856159 29877 heartbeater.cc:511] Master 127.12.8.62:38713 requested a full tablet report, sending...
I20260925 15:17:36.857650 29421 ts_manager.cc:194] Registered new tserver with Master: 220dbeb2675142febdc8769bf9f71b97 (127.12.8.3:33727)
I20260925 15:17:36.858606 29421 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.3@KRBTEST.COM'} at 127.12.8.3:58851
I20260925 15:17:36.861258 12320 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:17:36.877677 29612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:36.865702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55228 (local address 127.12.8.62:38713)
0925 15:17:36.865907 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:36.865916 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:36.866350 (+ 434us) server_negotiation.cc:407] Connection header received
0925 15:17:36.866485 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:36.866488 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:36.866547 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:36.866624 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:17:36.867909 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:36.868614 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:36.869886 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:36.870431 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:36.870528 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:36.872900 (+ 2372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:17:36.872905 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:17:36.872908 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:17:36.872936 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:17:36.875365 (+ 2429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:17:36.875976 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:17:36.875980 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:17:36.875982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:17:36.876094 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:17:36.876402 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:17:36.876405 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:17:36.876407 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:17:36.876691 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:17:36.876876 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:36.877158 (+ 282us) server_negotiation.cc:299] Negotiation successful
0925 15:17:36.877331 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 15:17:36.881816 29421 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55228:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:17:36.888706 29421 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:17:36.927184 29891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:36.901082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36786 (local address 127.12.8.3:33727)
0925 15:17:36.918946 (+ 17864us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:36.918955 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:36.918982 (+ 27us) server_negotiation.cc:407] Connection header received
0925 15:17:36.919061 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:36.919067 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:36.919286 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:36.919492 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:17:36.920669 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:36.922295 (+ 1626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:36.925041 (+ 2746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:36.925938 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:36.926090 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:36.926357 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:36.926647 (+ 290us) server_negotiation.cc:299] Negotiation successful
0925 15:17:36.926821 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17672,"thread_start_us":17495,"threads_started":1}
I20260925 15:17:36.927644 29889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:36.900849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.3:33727 (local address 127.0.0.1:36786)
0925 15:17:36.916380 (+ 15531us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:36.916431 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 15:17:36.916609 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:17:36.919768 (+ 3159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:17:36.919773 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:17:36.919810 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:17:36.920431 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:36.920447 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:36.922422 (+ 1975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:36.922431 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:36.924898 (+ 2467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:36.924907 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:36.926139 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:36.926145 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:36.926390 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:36.926440 (+ 50us) client_negotiation.cc:769] Sending connection context
0925 15:17:36.926615 (+ 175us) client_negotiation.cc:240] Negotiation successful
0925 15:17:36.926811 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15284,"mutex_wait_us":178,"thread_start_us":15136,"threads_started":1}
I20260925 15:17:36.927968 29894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:36.903159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45446 (local address 127.12.8.1:43981)
0925 15:17:36.919324 (+ 16165us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:36.919332 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:36.919359 (+ 27us) server_negotiation.cc:407] Connection header received
0925 15:17:36.919457 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:36.919464 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:36.919697 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:36.919899 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:17:36.920680 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:36.921675 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:36.924523 (+ 2848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:36.926036 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:36.926219 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:36.926448 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:36.927498 (+ 1050us) server_negotiation.cc:299] Negotiation successful
0925 15:17:36.927664 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15981,"thread_start_us":16014,"threads_started":1}
I20260925 15:17:36.928164 29890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:36.900898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.2:42297 (local address 127.0.0.1:36330)
0925 15:17:36.918554 (+ 17656us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:36.918581 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 15:17:36.918727 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:17:36.919618 (+ 891us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:17:36.919625 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:17:36.919652 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:17:36.920184 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:36.920196 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:36.922141 (+ 1945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:36.922145 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:36.924604 (+ 2459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:36.924618 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:36.926614 (+ 1996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:36.926619 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:36.926962 (+ 343us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:36.927038 (+ 76us) client_negotiation.cc:769] Sending connection context
0925 15:17:36.927125 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 15:17:36.927250 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17508,"mutex_wait_us":143,"thread_start_us":17376,"threads_started":1}
I20260925 15:17:36.928237 29892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:36.901499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36330 (local address 127.12.8.2:42297)
0925 15:17:36.915369 (+ 13870us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:36.915383 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:36.918883 (+ 3500us) server_negotiation.cc:407] Connection header received
0925 15:17:36.919008 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:36.919018 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:36.919286 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:36.919495 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:17:36.920386 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:36.922009 (+ 1623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:36.924761 (+ 2752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:36.926124 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:36.926256 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:36.926409 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:36.927658 (+ 1249us) server_negotiation.cc:299] Negotiation successful
0925 15:17:36.927824 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13670,"thread_start_us":13535,"threads_started":1}
I20260925 15:17:36.928623 29893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:36.902616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.1:43981 (local address 127.0.0.1:45446)
0925 15:17:36.918953 (+ 16337us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:36.918987 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 15:17:36.919105 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:17:36.919905 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:17:36.919909 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:17:36.919938 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:17:36.920504 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:36.920513 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:36.921823 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:36.921831 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:36.924364 (+ 2533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:36.924376 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:36.926612 (+ 2236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:36.926618 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:36.926954 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:36.927036 (+ 82us) client_negotiation.cc:769] Sending connection context
0925 15:17:36.927136 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 15:17:36.928310 (+ 1174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16177,"mutex_wait_us":207,"thread_start_us":16766,"threads_started":1}
I20260925 15:17:36.931922 29679 tablet_service.cc:1467] Processing CreateTablet for tablet 97687392947e453ab1d0e3b9157740e4 (DEFAULT_TABLE table=test-table [id=44ccab34790c4e9aa64d6a40b87ef6e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:17:36.931927 29812 tablet_service.cc:1467] Processing CreateTablet for tablet 97687392947e453ab1d0e3b9157740e4 (DEFAULT_TABLE table=test-table [id=44ccab34790c4e9aa64d6a40b87ef6e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:17:36.933184 29679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97687392947e453ab1d0e3b9157740e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:17:36.933312 29812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97687392947e453ab1d0e3b9157740e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:17:36.933386 29545 tablet_service.cc:1467] Processing CreateTablet for tablet 97687392947e453ab1d0e3b9157740e4 (DEFAULT_TABLE table=test-table [id=44ccab34790c4e9aa64d6a40b87ef6e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:17:36.934753 29545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97687392947e453ab1d0e3b9157740e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:17:36.944527 29895 tablet_bootstrap.cc:492] T 97687392947e453ab1d0e3b9157740e4 P 3f98e52717de46769158682003c72b55: Bootstrap starting.
I20260925 15:17:36.947625 29896 tablet_bootstrap.cc:492] T 97687392947e453ab1d0e3b9157740e4 P 220dbeb2675142febdc8769bf9f71b97: Bootstrap starting.
I20260925 15:17:36.948189 29895 tablet_bootstrap.cc:654] T 97687392947e453ab1d0e3b9157740e4 P 3f98e52717de46769158682003c72b55: Neither blocks nor log segments found. Creating new log.
I20260925 15:17:36.949151 29897 tablet_bootstrap.cc:492] T 97687392947e453ab1d0e3b9157740e4 P 557bedf5a03c4cd696ae9129d1ef4339: Bootstrap starting.
I20260925 15:17:36.950244 29895 log.cc:824] T 97687392947e453ab1d0e3b9157740e4 P 3f98e52717de46769158682003c72b55: Log is configured to *not* fsync() on all Append() calls
I20260925 15:17:36.951392 29896 tablet_bootstrap.cc:654] T 97687392947e453ab1d0e3b9157740e4 P 220dbeb2675142febdc8769bf9f71b97: Neither blocks nor log segments found. Creating new log.
I20260925 15:17:36.952389 29896 log.cc:824] T 97687392947e453ab1d0e3b9157740e4 P 220dbeb2675142febdc8769bf9f71b97: Log is configured to *not* fsync() on all Append() calls
I20260925 15:17:36.952718 29897 tablet_bootstrap.cc:654] T 97687392947e453ab1d0e3b9157740e4 P 557bedf5a03c4cd696ae9129d1ef4339: Neither blocks nor log segments found. Creating new log.
I20260925 15:17:36.953534 29895 tablet_bootstrap.cc:492] T 97687392947e453ab1d0e3b9157740e4 P 3f98e52717de46769158682003c72b55: No bootstrap required, opened a new log
I20260925 15:17:36.953856 29895 ts_tablet_manager.cc:1397] T 97687392947e453ab1d0e3b9157740e4 P 3f98e52717de46769158682003c72b55: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260925 15:17:36.953935 29897 log.cc:824] T 97687392947e453ab1d0e3b9157740e4 P 557bedf5a03c4cd696ae9129d1ef4339: Log is configured to *not* fsync() on all Append() calls
I20260925 15:17:36.955309 29896 tablet_bootstrap.cc:492] T 97687392947e453ab1d0e3b9157740e4 P 220dbeb2675142febdc8769bf9f71b97: No bootstrap required, opened a new log
I20260925 15:17:36.955576 29896 ts_tablet_manager.cc:1397] T 97687392947e453ab1d0e3b9157740e4 P 220dbeb2675142febdc8769bf9f71b97: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.001s
I20260925 15:17:36.956782 29897 tablet_bootstrap.cc:492] T 97687392947e453ab1d0e3b9157740e4 P 557bedf5a03c4cd696ae9129d1ef4339: No bootstrap required, opened a new log
I20260925 15:17:36.957044 29897 ts_tablet_manager.cc:1397] T 97687392947e453ab1d0e3b9157740e4 P 557bedf5a03c4cd696ae9129d1ef4339: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260925 15:17:36.963394 29896 raft_consensus.cc:348] T 97687392947e453ab1d0e3b9157740e4 P 220dbeb2675142febdc8769bf9f71b97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220dbeb2675142febdc8769bf9f71b97" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 33727 } } peers { permanent_uuid: "3f98e52717de46769158682003c72b55" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 42297 } } peers { permanent_uuid: "557bedf5a03c4cd696ae9129d1ef4339" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 43981 } }
I20260925 15:17:36.963923 29896 raft_consensus.cc:374] T 97687392947e453ab1d0e3b9157740e4 P 220dbeb2675142febdc8769bf9f71b97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:17:36.964028 29896 raft_consensus.cc:729] T 97687392947e453ab1d0e3b9157740e4 P 220dbeb2675142febdc8769bf9f71b97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 220dbeb2675142febdc8769bf9f71b97, State: Initialized, Role: FOLLOWER
I20260925 15:17:36.964562 29895 raft_consensus.cc:348] T 97687392947e453ab1d0e3b9157740e4 P 3f98e52717de46769158682003c72b55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220dbeb2675142febdc8769bf9f71b97" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 33727 } } peers { permanent_uuid: "3f98e52717de46769158682003c72b55" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 42297 } } peers { permanent_uuid: "557bedf5a03c4cd696ae9129d1ef4339" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 43981 } }
I20260925 15:17:36.964954 29895 raft_consensus.cc:374] T 97687392947e453ab1d0e3b9157740e4 P 3f98e52717de46769158682003c72b55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:17:36.964794 29896 consensus_queue.cc:260] T 97687392947e453ab1d0e3b9157740e4 P 220dbeb2675142febdc8769bf9f71b97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220dbeb2675142febdc8769bf9f71b97" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 33727 } } peers { permanent_uuid: "3f98e52717de46769158682003c72b55" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 42297 } } peers { permanent_uuid: "557bedf5a03c4cd696ae9129d1ef4339" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 43981 } }
I20260925 15:17:36.965054 29895 raft_consensus.cc:729] T 97687392947e453ab1d0e3b9157740e4 P 3f98e52717de46769158682003c72b55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f98e52717de46769158682003c72b55, State: Initialized, Role: FOLLOWER
I20260925 15:17:36.965633 29895 consensus_queue.cc:260] T 97687392947e453ab1d0e3b9157740e4 P 3f98e52717de46769158682003c72b55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220dbeb2675142febdc8769bf9f71b97" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 33727 } } peers { permanent_uuid: "3f98e52717de46769158682003c72b55" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 42297 } } peers { permanent_uuid: "557bedf5a03c4cd696ae9129d1ef4339" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 43981 } }
I20260925 15:17:36.966063 29877 heartbeater.cc:503] Master 127.12.8.62:38713 was elected leader, sending a full tablet report...
I20260925 15:17:36.966852 29744 heartbeater.cc:503] Master 127.12.8.62:38713 was elected leader, sending a full tablet report...
I20260925 15:17:36.967505 29895 ts_tablet_manager.cc:1428] T 97687392947e453ab1d0e3b9157740e4 P 3f98e52717de46769158682003c72b55: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260925 15:17:36.967999 29896 ts_tablet_manager.cc:1428] T 97687392947e453ab1d0e3b9157740e4 P 220dbeb2675142febdc8769bf9f71b97: Time spent starting tablet: real 0.012s user 0.003s sys 0.008s
I20260925 15:17:36.968542 29897 raft_consensus.cc:348] T 97687392947e453ab1d0e3b9157740e4 P 557bedf5a03c4cd696ae9129d1ef4339 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220dbeb2675142febdc8769bf9f71b97" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 33727 } } peers { permanent_uuid: "3f98e52717de46769158682003c72b55" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 42297 } } peers { permanent_uuid: "557bedf5a03c4cd696ae9129d1ef4339" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 43981 } }
I20260925 15:17:36.968878 29897 raft_consensus.cc:374] T 97687392947e453ab1d0e3b9157740e4 P 557bedf5a03c4cd696ae9129d1ef4339 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:17:36.968978 29897 raft_consensus.cc:729] T 97687392947e453ab1d0e3b9157740e4 P 557bedf5a03c4cd696ae9129d1ef4339 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 557bedf5a03c4cd696ae9129d1ef4339, State: Initialized, Role: FOLLOWER
I20260925 15:17:36.969578 29897 consensus_queue.cc:260] T 97687392947e453ab1d0e3b9157740e4 P 557bedf5a03c4cd696ae9129d1ef4339 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220dbeb2675142febdc8769bf9f71b97" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 33727 } } peers { permanent_uuid: "3f98e52717de46769158682003c72b55" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 42297 } } peers { permanent_uuid: "557bedf5a03c4cd696ae9129d1ef4339" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 43981 } }
I20260925 15:17:36.971216 29897 ts_tablet_manager.cc:1428] T 97687392947e453ab1d0e3b9157740e4 P 557bedf5a03c4cd696ae9129d1ef4339: Time spent starting tablet: real 0.014s user 0.013s sys 0.001s
W20260925 15:17:37.054855 29611 tablet.cc:2369] T 97687392947e453ab1d0e3b9157740e4 P 557bedf5a03c4cd696ae9129d1ef4339: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:17:37.068898 29745 tablet.cc:2369] T 97687392947e453ab1d0e3b9157740e4 P 3f98e52717de46769158682003c72b55: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:17:37.083248 29878 tablet.cc:2369] T 97687392947e453ab1d0e3b9157740e4 P 220dbeb2675142febdc8769bf9f71b97: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:17:37.119379 29901 raft_consensus.cc:482] T 97687392947e453ab1d0e3b9157740e4 P 220dbeb2675142febdc8769bf9f71b97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:17:37.119699 29901 raft_consensus.cc:504] T 97687392947e453ab1d0e3b9157740e4 P 220dbeb2675142febdc8769bf9f71b97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220dbeb2675142febdc8769bf9f71b97" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 33727 } } peers { permanent_uuid: "3f98e52717de46769158682003c72b55" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 42297 } } peers { permanent_uuid: "557bedf5a03c4cd696ae9129d1ef4339" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 43981 } }
I20260925 15:17:37.121340 29901 leader_election.cc:288] T 97687392947e453ab1d0e3b9157740e4 P 220dbeb2675142febdc8769bf9f71b97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f98e52717de46769158682003c72b55 (127.12.8.2:42297), 557bedf5a03c4cd696ae9129d1ef4339 (127.12.8.1:43981)
I20260925 15:17:37.127584 29894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:37.121854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.3:53265 (local address 127.12.8.1:43981)
0925 15:17:37.122049 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:37.122055 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:37.122078 (+ 23us) server_negotiation.cc:407] Connection header received
0925 15:17:37.122296 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:37.122305 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:37.122388 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:37.122500 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:17:37.122915 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:37.123885 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:37.125639 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:37.126729 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:37.126827 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:37.126897 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:37.127239 (+ 342us) server_negotiation.cc:299] Negotiation successful
0925 15:17:37.127320 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 15:17:37.127674 29879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:37.121631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.1:43981 (local address 127.12.8.3:53265)
0925 15:17:37.121925 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:37.121948 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 15:17:37.122090 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:17:37.122527 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:17:37.122530 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:17:37.122560 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:17:37.122753 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:37.122785 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:37.124004 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:37.124010 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:37.125495 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:37.125506 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:37.126880 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:37.126888 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:37.127076 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:37.127112 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 15:17:37.127211 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 15:17:37.127356 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144}
I20260925 15:17:37.128264 29892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:37.121862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.3:55295 (local address 127.12.8.2:42297)
0925 15:17:37.122049 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:37.122055 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:37.122716 (+ 661us) server_negotiation.cc:407] Connection header received
0925 15:17:37.122867 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:37.122872 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:37.122947 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:37.123014 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:17:37.123584 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:37.124459 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:37.126348 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:37.127300 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:37.127419 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:37.127514 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:37.127882 (+ 368us) server_negotiation.cc:299] Negotiation successful
0925 15:17:37.128021 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 15:17:37.128786 29904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:37.121713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.2:42297 (local address 127.12.8.3:55295)
0925 15:17:37.122583 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:37.122605 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 15:17:37.122730 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:17:37.123054 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:17:37.123058 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:17:37.123082 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:17:37.123432 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:37.123443 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:37.124610 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:37.124614 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:37.126209 (+ 1595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:37.126223 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:37.127497 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:37.127502 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:37.127709 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:37.127738 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 15:17:37.128427 (+ 689us) client_negotiation.cc:240] Negotiation successful
0925 15:17:37.128553 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":752,"thread_start_us":559,"threads_started":1}
I20260925 15:17:37.128825 29565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97687392947e453ab1d0e3b9157740e4" candidate_uuid: "220dbeb2675142febdc8769bf9f71b97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "557bedf5a03c4cd696ae9129d1ef4339" is_pre_election: true
I20260925 15:17:37.129421 29565 raft_consensus.cc:2454] T 97687392947e453ab1d0e3b9157740e4 P 557bedf5a03c4cd696ae9129d1ef4339 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 220dbeb2675142febdc8769bf9f71b97 in term 0.
I20260925 15:17:37.130059 29766 leader_election.cc:302] T 97687392947e453ab1d0e3b9157740e4 P 220dbeb2675142febdc8769bf9f71b97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 220dbeb2675142febdc8769bf9f71b97, 557bedf5a03c4cd696ae9129d1ef4339; no voters:
I20260925 15:17:37.129973 29699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97687392947e453ab1d0e3b9157740e4" candidate_uuid: "220dbeb2675142febdc8769bf9f71b97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f98e52717de46769158682003c72b55" is_pre_election: true
I20260925 15:17:37.130481 29901 raft_consensus.cc:2781] T 97687392947e453ab1d0e3b9157740e4 P 220dbeb2675142febdc8769bf9f71b97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:17:37.130513 29699 raft_consensus.cc:2454] T 97687392947e453ab1d0e3b9157740e4 P 3f98e52717de46769158682003c72b55 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 220dbeb2675142febdc8769bf9f71b97 in term 0.
I20260925 15:17:37.130656 29901 raft_consensus.cc:482] T 97687392947e453ab1d0e3b9157740e4 P 220dbeb2675142febdc8769bf9f71b97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:17:37.130756 29901 raft_consensus.cc:3037] T 97687392947e453ab1d0e3b9157740e4 P 220dbeb2675142febdc8769bf9f71b97 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:17:37.133445 29901 raft_consensus.cc:504] T 97687392947e453ab1d0e3b9157740e4 P 220dbeb2675142febdc8769bf9f71b97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220dbeb2675142febdc8769bf9f71b97" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 33727 } } peers { permanent_uuid: "3f98e52717de46769158682003c72b55" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 42297 } } peers { permanent_uuid: "557bedf5a03c4cd696ae9129d1ef4339" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 43981 } }
I20260925 15:17:37.134114 29901 leader_election.cc:288] T 97687392947e453ab1d0e3b9157740e4 P 220dbeb2675142febdc8769bf9f71b97 [CANDIDATE]: Term 1 election: Requested vote from peers 3f98e52717de46769158682003c72b55 (127.12.8.2:42297), 557bedf5a03c4cd696ae9129d1ef4339 (127.12.8.1:43981)
I20260925 15:17:37.134449 29699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97687392947e453ab1d0e3b9157740e4" candidate_uuid: "220dbeb2675142febdc8769bf9f71b97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f98e52717de46769158682003c72b55"
I20260925 15:17:37.134658 29699 raft_consensus.cc:3037] T 97687392947e453ab1d0e3b9157740e4 P 3f98e52717de46769158682003c72b55 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:17:37.134670 29565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97687392947e453ab1d0e3b9157740e4" candidate_uuid: "220dbeb2675142febdc8769bf9f71b97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "557bedf5a03c4cd696ae9129d1ef4339"
I20260925 15:17:37.134919 29565 raft_consensus.cc:3037] T 97687392947e453ab1d0e3b9157740e4 P 557bedf5a03c4cd696ae9129d1ef4339 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:17:37.137297 29699 raft_consensus.cc:2454] T 97687392947e453ab1d0e3b9157740e4 P 3f98e52717de46769158682003c72b55 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 220dbeb2675142febdc8769bf9f71b97 in term 1.
I20260925 15:17:37.137692 29768 leader_election.cc:302] T 97687392947e453ab1d0e3b9157740e4 P 220dbeb2675142febdc8769bf9f71b97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 220dbeb2675142febdc8769bf9f71b97, 3f98e52717de46769158682003c72b55; no voters:
I20260925 15:17:37.137913 29565 raft_consensus.cc:2454] T 97687392947e453ab1d0e3b9157740e4 P 557bedf5a03c4cd696ae9129d1ef4339 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 220dbeb2675142febdc8769bf9f71b97 in term 1.
I20260925 15:17:37.137965 29901 raft_consensus.cc:2781] T 97687392947e453ab1d0e3b9157740e4 P 220dbeb2675142febdc8769bf9f71b97 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:17:37.138518 29901 raft_consensus.cc:686] T 97687392947e453ab1d0e3b9157740e4 P 220dbeb2675142febdc8769bf9f71b97 [term 1 LEADER]: Becoming Leader. State: Replica: 220dbeb2675142febdc8769bf9f71b97, State: Running, Role: LEADER
I20260925 15:17:37.139031 29901 consensus_queue.cc:237] T 97687392947e453ab1d0e3b9157740e4 P 220dbeb2675142febdc8769bf9f71b97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220dbeb2675142febdc8769bf9f71b97" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 33727 } } peers { permanent_uuid: "3f98e52717de46769158682003c72b55" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 42297 } } peers { permanent_uuid: "557bedf5a03c4cd696ae9129d1ef4339" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 43981 } }
I20260925 15:17:37.143389 29421 catalog_manager.cc:5495] T 97687392947e453ab1d0e3b9157740e4 P 220dbeb2675142febdc8769bf9f71b97 reported cstate change: term changed from 0 to 1, leader changed from <none> to 220dbeb2675142febdc8769bf9f71b97 (127.12.8.3). New cstate: current_term: 1 leader_uuid: "220dbeb2675142febdc8769bf9f71b97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220dbeb2675142febdc8769bf9f71b97" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 33727 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3f98e52717de46769158682003c72b55" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 42297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "557bedf5a03c4cd696ae9129d1ef4339" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 43981 } health_report { overall_health: UNKNOWN } } }
I20260925 15:17:37.206394 29891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:37.200761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36798 (local address 127.12.8.3:33727)
0925 15:17:37.200971 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:37.200977 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:37.201079 (+ 102us) server_negotiation.cc:407] Connection header received
0925 15:17:37.201329 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:37.201351 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:37.201439 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:37.201588 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:17:37.201981 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:37.202753 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:37.204338 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:37.204718 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:37.204852 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:37.205329 (+ 477us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:17:37.205813 (+ 484us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:17:37.205968 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:37.206025 (+ 57us) server_negotiation.cc:299] Negotiation successful
0925 15:17:37.206105 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260925 15:17:37.214756 29699 raft_consensus.cc:1261] T 97687392947e453ab1d0e3b9157740e4 P 3f98e52717de46769158682003c72b55 [term 1 FOLLOWER]: Refusing update from remote peer 220dbeb2675142febdc8769bf9f71b97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:17:37.215806 29565 raft_consensus.cc:1261] T 97687392947e453ab1d0e3b9157740e4 P 557bedf5a03c4cd696ae9129d1ef4339 [term 1 FOLLOWER]: Refusing update from remote peer 220dbeb2675142febdc8769bf9f71b97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:17:37.215747 29905 consensus_queue.cc:1035] T 97687392947e453ab1d0e3b9157740e4 P 220dbeb2675142febdc8769bf9f71b97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f98e52717de46769158682003c72b55" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 42297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:17:37.216598 29905 consensus_queue.cc:1035] T 97687392947e453ab1d0e3b9157740e4 P 220dbeb2675142febdc8769bf9f71b97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "557bedf5a03c4cd696ae9129d1ef4339" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 43981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:17:37.227449 29908 mvcc.cc:205] Tried to move back new op lower bound from 7333271376741449728 to 7333271376445190144. Current Snapshot: MvccSnapshot[applied={T|T < 7333271376741449728}]
I20260925 15:17:37.242100 29420 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55228:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:17:37.242446 29420 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55228:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:17:37.246354 29420 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4addb03da683435f81e6cec78ccf2c15: Sending DeleteTablet for 3 replicas of tablet 97687392947e453ab1d0e3b9157740e4
I20260925 15:17:37.247546 29812 tablet_service.cc:1514] Processing DeleteTablet for tablet 97687392947e453ab1d0e3b9157740e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:17:37 UTC) from {username='kudu', principal='kudu/127.12.8.62@KRBTEST.COM'} at 127.0.0.1:36786
I20260925 15:17:37.247558 29679 tablet_service.cc:1514] Processing DeleteTablet for tablet 97687392947e453ab1d0e3b9157740e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:17:37 UTC) from {username='kudu', principal='kudu/127.12.8.62@KRBTEST.COM'} at 127.0.0.1:36330
I20260925 15:17:37.247763 29545 tablet_service.cc:1514] Processing DeleteTablet for tablet 97687392947e453ab1d0e3b9157740e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:17:37 UTC) from {username='kudu', principal='kudu/127.12.8.62@KRBTEST.COM'} at 127.0.0.1:45446
I20260925 15:17:37.250276 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 29482
I20260925 15:17:37.266305 29920 tablet_replica.cc:330] T 97687392947e453ab1d0e3b9157740e4 P 557bedf5a03c4cd696ae9129d1ef4339: stopping tablet replica
I20260925 15:17:37.266857 29919 tablet_replica.cc:330] T 97687392947e453ab1d0e3b9157740e4 P 3f98e52717de46769158682003c72b55: stopping tablet replica
I20260925 15:17:37.267475 29918 tablet_replica.cc:330] T 97687392947e453ab1d0e3b9157740e4 P 220dbeb2675142febdc8769bf9f71b97: stopping tablet replica
I20260925 15:17:37.267517 29920 raft_consensus.cc:2229] T 97687392947e453ab1d0e3b9157740e4 P 557bedf5a03c4cd696ae9129d1ef4339 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:17:37.267874 29918 raft_consensus.cc:2229] T 97687392947e453ab1d0e3b9157740e4 P 220dbeb2675142febdc8769bf9f71b97 [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:17:37.267943 29919 raft_consensus.cc:2229] T 97687392947e453ab1d0e3b9157740e4 P 3f98e52717de46769158682003c72b55 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:17:37.268496 29919 raft_consensus.cc:2258] T 97687392947e453ab1d0e3b9157740e4 P 3f98e52717de46769158682003c72b55 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:17:37.268532 29918 raft_consensus.cc:2258] T 97687392947e453ab1d0e3b9157740e4 P 220dbeb2675142febdc8769bf9f71b97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:17:37.269191 29920 raft_consensus.cc:2258] T 97687392947e453ab1d0e3b9157740e4 P 557bedf5a03c4cd696ae9129d1ef4339 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:17:37.271252 29918 ts_tablet_manager.cc:1905] T 97687392947e453ab1d0e3b9157740e4 P 220dbeb2675142febdc8769bf9f71b97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:17:37.272084 29920 ts_tablet_manager.cc:1905] T 97687392947e453ab1d0e3b9157740e4 P 557bedf5a03c4cd696ae9129d1ef4339: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:17:37.272923 29919 ts_tablet_manager.cc:1905] T 97687392947e453ab1d0e3b9157740e4 P 3f98e52717de46769158682003c72b55: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 25 15:17:37 dist-test-slave-p468 krb5kdc[29376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.1@KRBTEST.COM
I20260925 15:17:37.278493 29918 ts_tablet_manager.cc:1918] T 97687392947e453ab1d0e3b9157740e4 P 220dbeb2675142febdc8769bf9f71b97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:17:37.278755 29918 log.cc:1196] T 97687392947e453ab1d0e3b9157740e4 P 220dbeb2675142febdc8769bf9f71b97: Deleting WAL directory at /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349162370577-12320-0/minicluster-data/ts-2/wal/wals/97687392947e453ab1d0e3b9157740e4
I20260925 15:17:37.279377 29918 ts_tablet_manager.cc:1939] T 97687392947e453ab1d0e3b9157740e4 P 220dbeb2675142febdc8769bf9f71b97: Deleting consensus metadata
I20260925 15:17:37.281258 29407 catalog_manager.cc:4841] TS 220dbeb2675142febdc8769bf9f71b97 (127.12.8.3:33727): tablet 97687392947e453ab1d0e3b9157740e4 (table test-table [id=44ccab34790c4e9aa64d6a40b87ef6e1]) successfully deleted
I20260925 15:17:37.283043 29894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:37.251932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45454 (local address 127.12.8.1:43981)
0925 15:17:37.252157 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:37.252166 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:37.256192 (+ 4026us) server_negotiation.cc:407] Connection header received
0925 15:17:37.257075 (+ 883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:37.257082 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:37.257228 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:37.257320 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:17:37.262006 (+ 4686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:37.263592 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:37.268884 (+ 5292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:37.269679 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:37.269819 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:37.275423 (+ 5604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:17:37.275438 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:17:37.275454 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:17:37.275500 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:17:37.279545 (+ 4045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:17:37.280491 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:17:37.280501 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:17:37.280504 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:17:37.280662 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:17:37.281125 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:17:37.281132 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:17:37.281138 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:17:37.281575 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:17:37.281866 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:37.282444 (+ 578us) server_negotiation.cc:299] Negotiation successful
0925 15:17:37.282665 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 15:17:37.283677 29605 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:17:37.283736 29919 ts_tablet_manager.cc:1918] T 97687392947e453ab1d0e3b9157740e4 P 3f98e52717de46769158682003c72b55: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:17:37.283892 29919 log.cc:1196] T 97687392947e453ab1d0e3b9157740e4 P 3f98e52717de46769158682003c72b55: Deleting WAL directory at /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349162370577-12320-0/minicluster-data/ts-1/wal/wals/97687392947e453ab1d0e3b9157740e4
I20260925 15:17:37.284365 29919 ts_tablet_manager.cc:1939] T 97687392947e453ab1d0e3b9157740e4 P 3f98e52717de46769158682003c72b55: Deleting consensus metadata
I20260925 15:17:37.285831 29408 catalog_manager.cc:4841] TS 3f98e52717de46769158682003c72b55 (127.12.8.2:42297): tablet 97687392947e453ab1d0e3b9157740e4 (table test-table [id=44ccab34790c4e9aa64d6a40b87ef6e1]) successfully deleted
I20260925 15:17:37.375056 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 29482
I20260925 15:17:37.397176 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 29616
W20260925 15:17:37.397215 29406 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.8.1:43981 (error 108)
I20260925 15:17:37.399385 29893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:37.398860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.1:43981 (local address 127.0.0.1:45458)
0925 15:17:37.399101 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:37.399215 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.1:43981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
W20260925 15:17:37.399626 29406 catalog_manager.cc:4584] TS 557bedf5a03c4cd696ae9129d1ef4339 (127.12.8.1:43981): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 97687392947e453ab1d0e3b9157740e4: Network error: Client connection negotiation failed: client connection to 127.12.8.1:43981: connect: Connection refused (error 111)
Sep 25 15:17:37 dist-test-slave-p468 krb5kdc[29376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.2@KRBTEST.COM
I20260925 15:17:37.414693 29892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:37.398421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36336 (local address 127.12.8.2:42297)
0925 15:17:37.398596 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:37.398601 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:37.398622 (+ 21us) server_negotiation.cc:407] Connection header received
0925 15:17:37.398685 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:37.398689 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:37.398765 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:37.398889 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:17:37.400354 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:37.401092 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:37.402380 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:37.402771 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:37.402899 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:37.407753 (+ 4854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:17:37.407770 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:17:37.407791 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:17:37.407846 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:17:37.411707 (+ 3861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:17:37.412485 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:17:37.412494 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:17:37.412499 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:17:37.412666 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:17:37.413098 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:17:37.413104 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:17:37.413107 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:17:37.413591 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:17:37.413884 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:37.414088 (+ 204us) server_negotiation.cc:299] Negotiation successful
0925 15:17:37.414331 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 15:17:37.415424 29739 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:17:37.453208 29893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:37.452677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.1:43981 (local address 127.0.0.1:45472)
0925 15:17:37.452908 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:37.453016 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.1:43981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260925 15:17:37.454372 29893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:37.453947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.1:43981 (local address 127.0.0.1:45476)
0925 15:17:37.454107 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:37.454231 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.1:43981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260925 15:17:37.505759 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 29616
I20260925 15:17:37.515306 29893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:37.514725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.1:43981 (local address 127.0.0.1:45486)
0925 15:17:37.514969 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:37.515084 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.1:43981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260925 15:17:37.516597 29893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:37.516183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.1:43981 (local address 127.0.0.1:45488)
0925 15:17:37.516363 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:37.516451 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.1:43981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260925 15:17:37.522933 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 29749
Sep 25 15:17:37 dist-test-slave-p468 krb5kdc[29376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.3@KRBTEST.COM
I20260925 15:17:37.539911 29891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:37.523698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36808 (local address 127.12.8.3:33727)
0925 15:17:37.523888 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:37.523894 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:37.523917 (+ 23us) server_negotiation.cc:407] Connection header received
0925 15:17:37.523997 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:37.524003 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:37.524083 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:37.524191 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:17:37.525455 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:37.526762 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:37.527979 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:37.528662 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:37.528798 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:37.533819 (+ 5021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:17:37.533830 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:17:37.533846 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:17:37.533878 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:17:37.537039 (+ 3161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:17:37.537848 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:17:37.537855 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:17:37.537857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:17:37.538020 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:17:37.538473 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:17:37.538477 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:17:37.538479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:17:37.538812 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:17:37.539031 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:37.539382 (+ 351us) server_negotiation.cc:299] Negotiation successful
0925 15:17:37.539546 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 15:17:37.540612 29872 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:17:37.597239 29893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:37.596770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.1:43981 (local address 127.0.0.1:45498)
0925 15:17:37.596981 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:37.597074 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.1:43981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260925 15:17:37.598332 29893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:37.597876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.1:43981 (local address 127.0.0.1:45512)
0925 15:17:37.598062 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:37.598164 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.1:43981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260925 15:17:37.663087 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 29749
I20260925 15:17:37.678009 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 29389
I20260925 15:17:37.679171 29451 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:17:37.789227 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 29389
2026-09-25T15:17:37Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4375 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-25T15:17:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:17:37Z Disabled control of system clock
I20260925 15:17:37.869060 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349162370577-12320-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349162370577-12320-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349162370577-12320-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.62:40305
--webserver_interface=127.12.8.62
--webserver_port=0
--builtin_ntp_servers=127.12.8.20:42559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.62:40305
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 15:17:38.191489 29930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:17:38.191821 29930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:17:38.191967 29930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:17:38.199048 29930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:17:38.199186 29930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:17:38.199265 29930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:17:38.199373 29930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:17:38.199446 29930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:17:38.208398 29930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:42559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349162370577-12320-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349162370577-12320-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.8.62:40305
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.62:40305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.8.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349162370577-12320-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:17:38.209630 29930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:17:38.211439 29930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:17:38.222973 29936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:17:38.222990 29938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:17:38.223035 29935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:17:38.223796 29930 server_base.cc:1034] running on GCE node
I20260925 15:17:38.224604 29930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:17:38.226007 29930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:17:38.227212 29930 hybrid_clock.cc:648] HybridClock initialized: now 1790349458227160 us; error 42 us; skew 500 ppm
I20260925 15:17:38.227574 29930 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:17:38.232733 29930 webserver.cc:460] Webserver started at http://127.12.8.62:39257/ using document root <none> and password file <none>
I20260925 15:17:38.233456 29930 fs_manager.cc:359] Metadata directory not provided
I20260925 15:17:38.233575 29930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:17:38.233923 29930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:17:38.236908 29930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349162370577-12320-0/minicluster-data/master-0/data/instance:
uuid: "2f589af505144a87bf0ebabea4a3a5c4"
format_stamp: "Formatted at 2026-09-25 15:17:38 on dist-test-slave-p468"
server_key: "c1ca5541fb742c31ec09f36d0b52d165"
server_key_iv: "bc010f2a3603a3251c180e725fbafca8"
server_key_version: "encryptionkey@0"
I20260925 15:17:38.237730 29930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349162370577-12320-0/minicluster-data/master-0/wal/instance:
uuid: "2f589af505144a87bf0ebabea4a3a5c4"
format_stamp: "Formatted at 2026-09-25 15:17:38 on dist-test-slave-p468"
server_key: "c1ca5541fb742c31ec09f36d0b52d165"
server_key_iv: "bc010f2a3603a3251c180e725fbafca8"
server_key_version: "encryptionkey@0"
I20260925 15:17:38.242933 29930 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260925 15:17:38.246340 29944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:17:38.247627 29930 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 15:17:38.247800 29930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349162370577-12320-0/minicluster-data/master-0/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349162370577-12320-0/minicluster-data/master-0/wal
uuid: "2f589af505144a87bf0ebabea4a3a5c4"
format_stamp: "Formatted at 2026-09-25 15:17:38 on dist-test-slave-p468"
server_key: "c1ca5541fb742c31ec09f36d0b52d165"
server_key_iv: "bc010f2a3603a3251c180e725fbafca8"
server_key_version: "encryptionkey@0"
I20260925 15:17:38.248031 29930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349162370577-12320-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349162370577-12320-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349162370577-12320-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:17:38.273692 29930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:17:38.274623 29930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:17:38.274917 29930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:17:38.293715 29930 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.62:40305
I20260925 15:17:38.293727 29995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.62:40305 every 8 connection(s)
I20260925 15:17:38.295034 29930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
I20260925 15:17:38.298622 29996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:17:38.299463 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 29930
I20260925 15:17:38.299649 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349162370577-12320-0/minicluster-data/master-0/wal/instance
I20260925 15:17:38.300086 12320 external_mini_cluster.cc:1428] Setting key ebe07f6bd15e061bc623d9472178fb4f
I20260925 15:17:38.312917 29996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f589af505144a87bf0ebabea4a3a5c4: Bootstrap starting.
I20260925 15:17:38.315219 29999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:38.301993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56784 (local address 127.12.8.62:40305)
0925 15:17:38.303117 (+ 1124us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:38.303135 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:38.303186 (+ 51us) server_negotiation.cc:407] Connection header received
0925 15:17:38.305164 (+ 1978us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:38.305191 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:38.305797 (+ 606us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:38.306255 (+ 458us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:17:38.306779 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:38.308609 (+ 1830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:38.310004 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:38.311063 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:38.311226 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:38.312466 (+ 1240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:17:38.312475 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:17:38.312494 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:17:38.313175 (+ 681us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:17:38.313272 (+ 97us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:17:38.313297 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:17:38.313424 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:38.313643 (+ 219us) server_negotiation.cc:299] Negotiation successful
0925 15:17:38.313942 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":393,"threads_started":1}
I20260925 15:17:38.316597 29996 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f589af505144a87bf0ebabea4a3a5c4: Neither blocks nor log segments found. Creating new log.
I20260925 15:17:38.317644 29996 log.cc:824] T 00000000000000000000000000000000 P 2f589af505144a87bf0ebabea4a3a5c4: Log is configured to *not* fsync() on all Append() calls
I20260925 15:17:38.320291 29996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f589af505144a87bf0ebabea4a3a5c4: No bootstrap required, opened a new log
I20260925 15:17:38.327307 29996 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f589af505144a87bf0ebabea4a3a5c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f589af505144a87bf0ebabea4a3a5c4" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 40305 } }
I20260925 15:17:38.327695 29996 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f589af505144a87bf0ebabea4a3a5c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:17:38.327805 29996 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f589af505144a87bf0ebabea4a3a5c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f589af505144a87bf0ebabea4a3a5c4, State: Initialized, Role: FOLLOWER
I20260925 15:17:38.328449 29996 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f589af505144a87bf0ebabea4a3a5c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f589af505144a87bf0ebabea4a3a5c4" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 40305 } }
I20260925 15:17:38.328716 29996 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f589af505144a87bf0ebabea4a3a5c4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:17:38.328876 29996 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f589af505144a87bf0ebabea4a3a5c4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:17:38.329051 29996 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f589af505144a87bf0ebabea4a3a5c4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:17:38.331269 29996 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f589af505144a87bf0ebabea4a3a5c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f589af505144a87bf0ebabea4a3a5c4" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 40305 } }
I20260925 15:17:38.331773 29996 leader_election.cc:302] T 00000000000000000000000000000000 P 2f589af505144a87bf0ebabea4a3a5c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f589af505144a87bf0ebabea4a3a5c4; no voters:
I20260925 15:17:38.332343 29996 leader_election.cc:288] T 00000000000000000000000000000000 P 2f589af505144a87bf0ebabea4a3a5c4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:17:38.332520 30001 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f589af505144a87bf0ebabea4a3a5c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:17:38.333268 30001 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f589af505144a87bf0ebabea4a3a5c4 [term 1 LEADER]: Becoming Leader. State: Replica: 2f589af505144a87bf0ebabea4a3a5c4, State: Running, Role: LEADER
I20260925 15:17:38.333838 30001 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f589af505144a87bf0ebabea4a3a5c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f589af505144a87bf0ebabea4a3a5c4" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 40305 } }
I20260925 15:17:38.334477 29996 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f589af505144a87bf0ebabea4a3a5c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:17:38.336702 30002 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f589af505144a87bf0ebabea4a3a5c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2f589af505144a87bf0ebabea4a3a5c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f589af505144a87bf0ebabea4a3a5c4" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 40305 } } }
I20260925 15:17:38.337149 30002 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f589af505144a87bf0ebabea4a3a5c4 [sys.catalog]: This master's current role is: LEADER
I20260925 15:17:38.337155 30003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f589af505144a87bf0ebabea4a3a5c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f589af505144a87bf0ebabea4a3a5c4. Latest consensus state: current_term: 1 leader_uuid: "2f589af505144a87bf0ebabea4a3a5c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f589af505144a87bf0ebabea4a3a5c4" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 40305 } } }
I20260925 15:17:38.337769 30003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f589af505144a87bf0ebabea4a3a5c4 [sys.catalog]: This master's current role is: LEADER
I20260925 15:17:38.339322 30008 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:17:38.344496 30008 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:17:38.352993 30008 catalog_manager.cc:1347] Generated new cluster ID: ed921b0628f14daa852d1d8a2a0aa9e6
I20260925 15:17:38.353099 30008 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:17:38.363724 30008 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:17:38.364944 30008 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:17:38.377887 30008 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f589af505144a87bf0ebabea4a3a5c4: Generated new TSK 0
I20260925 15:17:38.378757 30008 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:17:38.387656 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349162370577-12320-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349162370577-12320-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349162370577-12320-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349162370577-12320-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.1:0
--local_ip_for_outbound_sockets=127.12.8.1
--webserver_interface=127.12.8.1
--webserver_port=0
--tserver_master_addrs=127.12.8.62:40305
--builtin_ntp_servers=127.12.8.20:42559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 15:17:38.720783 30020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:17:38.721043 30020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:17:38.721130 30020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:17:38.728474 30020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:17:38.728566 30020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:17:38.728668 30020 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.1
I20260925 15:17:38.737757 30020 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:42559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349162370577-12320-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349162370577-12320-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349162370577-12320-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.8.1
--webserver_port=0
--tserver_master_addrs=127.12.8.62:40305
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.1
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349162370577-12320-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:17:38.739161 30020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:17:38.741114 30020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:17:38.754155 30025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:17:38.754179 30026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:17:38.754439 30028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:17:38.755209 30020 server_base.cc:1034] running on GCE node
I20260925 15:17:38.755731 30020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:17:38.756587 30020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:17:38.757824 30020 hybrid_clock.cc:648] HybridClock initialized: now 1790349458757791 us; error 55 us; skew 500 ppm
I20260925 15:17:38.758150 30020 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:17:38.760409 30020 webserver.cc:460] Webserver started at http://127.12.8.1:35257/ using document root <none> and password file <none>
I20260925 15:17:38.761083 30020 fs_manager.cc:359] Metadata directory not provided
I20260925 15:17:38.761217 30020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:17:38.761528 30020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:17:38.764371 30020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349162370577-12320-0/minicluster-data/ts-0/data/instance:
uuid: "f5c0ef93dbcb4dc99035de8ec45b0f9b"
format_stamp: "Formatted at 2026-09-25 15:17:38 on dist-test-slave-p468"
server_key: "bdc3a1b9f1e5f307bf837894a42ab4c9"
server_key_iv: "910b50a34e57730bfa5ce172c94076df"
server_key_version: "encryptionkey@0"
I20260925 15:17:38.765134 30020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349162370577-12320-0/minicluster-data/ts-0/wal/instance:
uuid: "f5c0ef93dbcb4dc99035de8ec45b0f9b"
format_stamp: "Formatted at 2026-09-25 15:17:38 on dist-test-slave-p468"
server_key: "bdc3a1b9f1e5f307bf837894a42ab4c9"
server_key_iv: "910b50a34e57730bfa5ce172c94076df"
server_key_version: "encryptionkey@0"
I20260925 15:17:38.770025 30020 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 15:17:38.773473 30034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:17:38.774886 30020 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:17:38.775060 30020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349162370577-12320-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349162370577-12320-0/minicluster-data/ts-0/wal
uuid: "f5c0ef93dbcb4dc99035de8ec45b0f9b"
format_stamp: "Formatted at 2026-09-25 15:17:38 on dist-test-slave-p468"
server_key: "bdc3a1b9f1e5f307bf837894a42ab4c9"
server_key_iv: "910b50a34e57730bfa5ce172c94076df"
server_key_version: "encryptionkey@0"
I20260925 15:17:38.775276 30020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349162370577-12320-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349162370577-12320-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349162370577-12320-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:17:38.791740 30020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:17:38.792502 30020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:17:38.792780 30020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:17:38.793670 30020 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:17:38.795213 30020 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:17:38.795336 30020 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:17:38.795469 30020 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:17:38.795559 30020 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:17:38.828506 30020 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.1:36335
I20260925 15:17:38.828593 30146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.1:36335 every 8 connection(s)
I20260925 15:17:38.829821 30020 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349162370577-12320-0/minicluster-data/ts-0/data/info.pb
I20260925 15:17:38.835753 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 30020
I20260925 15:17:38.835975 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349162370577-12320-0/minicluster-data/ts-0/wal/instance
I20260925 15:17:38.836429 12320 external_mini_cluster.cc:1428] Setting key 97e98b93dbcfd92d95a952be8e009ee3
I20260925 15:17:38.839232 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349162370577-12320-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349162370577-12320-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349162370577-12320-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349162370577-12320-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.2:0
--local_ip_for_outbound_sockets=127.12.8.2
--webserver_interface=127.12.8.2
--webserver_port=0
--tserver_master_addrs=127.12.8.62:40305
--builtin_ntp_servers=127.12.8.20:42559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 15:17:38.843465 30149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:38.832607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.1:50953 (local address 127.12.8.62:40305)
0925 15:17:38.833054 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:38.833061 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:38.834257 (+ 1196us) server_negotiation.cc:407] Connection header received
0925 15:17:38.836095 (+ 1838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:38.836101 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:38.836198 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:38.836347 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:17:38.837499 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:38.838463 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:38.840535 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:38.841068 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:38.841219 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:38.842619 (+ 1400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:17:38.842625 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:17:38.842630 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:17:38.842656 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:17:38.842701 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:17:38.842713 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:17:38.842814 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:38.843028 (+ 214us) server_negotiation.cc:299] Negotiation successful
0925 15:17:38.843119 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":207,"threads_started":1}
I20260925 15:17:38.844399 30150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:38.833214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:40305 (local address 127.12.8.1:50953)
0925 15:17:38.834058 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:38.834098 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 15:17:38.835787 (+ 1689us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:17:38.836577 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:17:38.836586 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:17:38.836668 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:17:38.837318 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:38.837343 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:38.838610 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:38.838616 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:38.840399 (+ 1783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:38.840411 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:38.841339 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:38.841346 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:38.841571 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:38.842310 (+ 739us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:17:38.842312 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:17:38.842351 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:17:38.842428 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:17:38.842812 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:17:38.842824 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:17:38.842834 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 15:17:38.843019 (+ 185us) client_negotiation.cc:240] Negotiation successful
0925 15:17:38.843401 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":234,"threads_started":1}
I20260925 15:17:38.845911 30147 heartbeater.cc:347] Connected to a master server at 127.12.8.62:40305
I20260925 15:17:38.846261 30147 heartbeater.cc:464] Registering TS with master...
I20260925 15:17:38.847180 30147 heartbeater.cc:511] Master 127.12.8.62:40305 requested a full tablet report, sending...
I20260925 15:17:38.849128 29961 ts_manager.cc:194] Registered new tserver with Master: f5c0ef93dbcb4dc99035de8ec45b0f9b (127.12.8.1:36335)
I20260925 15:17:38.850630 29961 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.8.1:50953
W20260925 15:17:39.161230 30151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:17:39.161557 30151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:17:39.161726 30151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:17:39.168869 30151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:17:39.169003 30151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:17:39.169178 30151 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.2
I20260925 15:17:39.177927 30151 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:42559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349162370577-12320-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349162370577-12320-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349162370577-12320-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.8.2
--webserver_port=0
--tserver_master_addrs=127.12.8.62:40305
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.2
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349162370577-12320-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:17:39.179261 30151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:17:39.180989 30151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:17:39.193995 30156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:17:39.194067 30157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:17:39.194495 30159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:17:39.194553 30151 server_base.cc:1034] running on GCE node
I20260925 15:17:39.195254 30151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:17:39.196103 30151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:17:39.197294 30151 hybrid_clock.cc:648] HybridClock initialized: now 1790349459197235 us; error 46 us; skew 500 ppm
I20260925 15:17:39.197592 30151 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:17:39.199810 30151 webserver.cc:460] Webserver started at http://127.12.8.2:36697/ using document root <none> and password file <none>
I20260925 15:17:39.200449 30151 fs_manager.cc:359] Metadata directory not provided
I20260925 15:17:39.200562 30151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:17:39.200865 30151 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:17:39.203593 30151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349162370577-12320-0/minicluster-data/ts-1/data/instance:
uuid: "bbe8e3a6eaf54ceb8f34d33a371a669d"
format_stamp: "Formatted at 2026-09-25 15:17:39 on dist-test-slave-p468"
server_key: "b9f88c2ce0e82f5d26c6362fb9001d71"
server_key_iv: "b1f4be847c60239c90f61bf43002086f"
server_key_version: "encryptionkey@0"
I20260925 15:17:39.204329 30151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349162370577-12320-0/minicluster-data/ts-1/wal/instance:
uuid: "bbe8e3a6eaf54ceb8f34d33a371a669d"
format_stamp: "Formatted at 2026-09-25 15:17:39 on dist-test-slave-p468"
server_key: "b9f88c2ce0e82f5d26c6362fb9001d71"
server_key_iv: "b1f4be847c60239c90f61bf43002086f"
server_key_version: "encryptionkey@0"
I20260925 15:17:39.209076 30151 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260925 15:17:39.212293 30165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:17:39.213495 30151 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 15:17:39.213670 30151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349162370577-12320-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349162370577-12320-0/minicluster-data/ts-1/wal
uuid: "bbe8e3a6eaf54ceb8f34d33a371a669d"
format_stamp: "Formatted at 2026-09-25 15:17:39 on dist-test-slave-p468"
server_key: "b9f88c2ce0e82f5d26c6362fb9001d71"
server_key_iv: "b1f4be847c60239c90f61bf43002086f"
server_key_version: "encryptionkey@0"
I20260925 15:17:39.213871 30151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349162370577-12320-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349162370577-12320-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349162370577-12320-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:17:39.239120 30151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:17:39.240072 30151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:17:39.240348 30151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:17:39.241382 30151 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:17:39.242877 30151 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:17:39.242995 30151 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:17:39.243124 30151 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:17:39.243214 30151 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:17:39.275207 30151 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.2:37955
I20260925 15:17:39.275292 30277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.2:37955 every 8 connection(s)
I20260925 15:17:39.276497 30151 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349162370577-12320-0/minicluster-data/ts-1/data/info.pb
I20260925 15:17:39.283541 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 30151
I20260925 15:17:39.283726 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349162370577-12320-0/minicluster-data/ts-1/wal/instance
I20260925 15:17:39.284181 12320 external_mini_cluster.cc:1428] Setting key 93d2a606cac205770cec1c05932a375b
I20260925 15:17:39.286854 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349162370577-12320-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349162370577-12320-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349162370577-12320-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349162370577-12320-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.3:0
--local_ip_for_outbound_sockets=127.12.8.3
--webserver_interface=127.12.8.3
--webserver_port=0
--tserver_master_addrs=127.12.8.62:40305
--builtin_ntp_servers=127.12.8.20:42559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 15:17:39.289139 30149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:39.279193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.2:41889 (local address 127.12.8.62:40305)
0925 15:17:39.279375 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:39.279381 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:39.280810 (+ 1429us) server_negotiation.cc:407] Connection header received
0925 15:17:39.282580 (+ 1770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:39.282585 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:39.282648 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:39.282733 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:17:39.283786 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:39.284578 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:39.286465 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:39.286857 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:39.286999 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:39.288346 (+ 1347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:17:39.288351 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:17:39.288354 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:17:39.288371 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:17:39.288402 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:17:39.288415 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:17:39.288513 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:39.288728 (+ 215us) server_negotiation.cc:299] Negotiation successful
0925 15:17:39.288826 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 15:17:39.290108 30280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:39.279778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:40305 (local address 127.12.8.2:41889)
0925 15:17:39.280644 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:39.280685 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 15:17:39.282339 (+ 1654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:17:39.282924 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:17:39.282932 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:17:39.283021 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:17:39.283579 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:39.283595 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:39.284851 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:39.284857 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:39.286312 (+ 1455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:39.286325 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:39.287080 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:39.287086 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:39.287309 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:39.288016 (+ 707us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:17:39.288019 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:17:39.288057 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:17:39.288135 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:17:39.288511 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:17:39.288523 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:17:39.288533 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 15:17:39.288716 (+ 183us) client_negotiation.cc:240] Negotiation successful
0925 15:17:39.289104 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":256,"threads_started":1}
I20260925 15:17:39.291662 30278 heartbeater.cc:347] Connected to a master server at 127.12.8.62:40305
I20260925 15:17:39.291952 30278 heartbeater.cc:464] Registering TS with master...
I20260925 15:17:39.292620 30278 heartbeater.cc:511] Master 127.12.8.62:40305 requested a full tablet report, sending...
I20260925 15:17:39.294166 29961 ts_manager.cc:194] Registered new tserver with Master: bbe8e3a6eaf54ceb8f34d33a371a669d (127.12.8.2:37955)
I20260925 15:17:39.295001 29961 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.8.2:41889
W20260925 15:17:39.619985 30281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:17:39.620249 30281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:17:39.620332 30281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:17:39.627444 30281 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:17:39.627548 30281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:17:39.627653 30281 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.3
I20260925 15:17:39.637301 30281 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:42559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349162370577-12320-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349162370577-12320-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349162370577-12320-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.8.3
--webserver_port=0
--tserver_master_addrs=127.12.8.62:40305
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.3
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349162370577-12320-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:17:39.638620 30281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:17:39.640466 30281 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:17:39.654675 30286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:17:39.654980 30287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:17:39.655054 30289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:17:39.656060 30281 server_base.cc:1034] running on GCE node
I20260925 15:17:39.656646 30281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:17:39.657648 30281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:17:39.658895 30281 hybrid_clock.cc:648] HybridClock initialized: now 1790349459658827 us; error 51 us; skew 500 ppm
I20260925 15:17:39.659245 30281 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:17:39.661428 30281 webserver.cc:460] Webserver started at http://127.12.8.3:41685/ using document root <none> and password file <none>
I20260925 15:17:39.662124 30281 fs_manager.cc:359] Metadata directory not provided
I20260925 15:17:39.662276 30281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:17:39.662602 30281 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:17:39.665465 30281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349162370577-12320-0/minicluster-data/ts-2/data/instance:
uuid: "a69933091d2747cfbb24de6e68395b21"
format_stamp: "Formatted at 2026-09-25 15:17:39 on dist-test-slave-p468"
server_key: "991a266a3ad9cfb173a11d3d2b478dfc"
server_key_iv: "77ca069ec9fd38987069aa02782d9a3b"
server_key_version: "encryptionkey@0"
I20260925 15:17:39.666320 30281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349162370577-12320-0/minicluster-data/ts-2/wal/instance:
uuid: "a69933091d2747cfbb24de6e68395b21"
format_stamp: "Formatted at 2026-09-25 15:17:39 on dist-test-slave-p468"
server_key: "991a266a3ad9cfb173a11d3d2b478dfc"
server_key_iv: "77ca069ec9fd38987069aa02782d9a3b"
server_key_version: "encryptionkey@0"
I20260925 15:17:39.671337 30281 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260925 15:17:39.674876 30295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:17:39.676152 30281 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 15:17:39.676349 30281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349162370577-12320-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349162370577-12320-0/minicluster-data/ts-2/wal
uuid: "a69933091d2747cfbb24de6e68395b21"
format_stamp: "Formatted at 2026-09-25 15:17:39 on dist-test-slave-p468"
server_key: "991a266a3ad9cfb173a11d3d2b478dfc"
server_key_iv: "77ca069ec9fd38987069aa02782d9a3b"
server_key_version: "encryptionkey@0"
I20260925 15:17:39.676571 30281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349162370577-12320-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349162370577-12320-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349162370577-12320-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:17:39.689496 30281 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:17:39.690366 30281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:17:39.690640 30281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:17:39.691650 30281 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:17:39.693068 30281 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:17:39.693186 30281 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:17:39.693316 30281 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:17:39.693408 30281 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:17:39.725673 30281 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.3:35451
I20260925 15:17:39.725728 30407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.3:35451 every 8 connection(s)
I20260925 15:17:39.727140 30281 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349162370577-12320-0/minicluster-data/ts-2/data/info.pb
I20260925 15:17:39.736865 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 30281
I20260925 15:17:39.737043 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349162370577-12320-0/minicluster-data/ts-2/wal/instance
I20260925 15:17:39.737511 12320 external_mini_cluster.cc:1428] Setting key b3300c4010f3e59b598b3717016da7d6
I20260925 15:17:39.739656 30149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:39.729533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.3:41615 (local address 127.12.8.62:40305)
0925 15:17:39.729721 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:39.729725 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:39.731095 (+ 1370us) server_negotiation.cc:407] Connection header received
0925 15:17:39.732879 (+ 1784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:39.732886 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:39.732980 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:39.733092 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:17:39.734160 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:39.734982 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:39.736611 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:39.737073 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:39.737218 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:39.738668 (+ 1450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:17:39.738674 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:17:39.738677 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:17:39.738694 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:17:39.738726 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:17:39.738738 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:17:39.738841 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:39.739056 (+ 215us) server_negotiation.cc:299] Negotiation successful
0925 15:17:39.739204 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 15:17:39.740741 30410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:39.730011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:40305 (local address 127.12.8.3:41615)
0925 15:17:39.730938 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:39.730979 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 15:17:39.732619 (+ 1640us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:17:39.733329 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:17:39.733337 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:17:39.733421 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:17:39.733961 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:39.733976 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:39.735129 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:39.735135 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:39.736440 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:39.736451 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:39.737311 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:39.737320 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:39.737546 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:39.738318 (+ 772us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:17:39.738321 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:17:39.738378 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:17:39.738462 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:17:39.738839 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:17:39.738852 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:17:39.738862 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 15:17:39.739047 (+ 185us) client_negotiation.cc:240] Negotiation successful
0925 15:17:39.739433 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":314,"threads_started":1}
I20260925 15:17:39.742009 30408 heartbeater.cc:347] Connected to a master server at 127.12.8.62:40305
I20260925 15:17:39.742332 30408 heartbeater.cc:464] Registering TS with master...
I20260925 15:17:39.743003 30408 heartbeater.cc:511] Master 127.12.8.62:40305 requested a full tablet report, sending...
I20260925 15:17:39.744266 29961 ts_manager.cc:194] Registered new tserver with Master: a69933091d2747cfbb24de6e68395b21 (127.12.8.3:35451)
I20260925 15:17:39.745041 29961 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.8.3:41615
I20260925 15:17:39.752593 12320 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:17:39.762555 30149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:39.756799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56800 (local address 127.12.8.62:40305)
0925 15:17:39.757012 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:39.757021 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:39.757464 (+ 443us) server_negotiation.cc:407] Connection header received
0925 15:17:39.757592 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:39.757596 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:39.757668 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:39.757750 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:17:39.758244 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:39.758995 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:39.760254 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:39.760640 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:39.760735 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:39.761767 (+ 1032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:17:39.761771 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:17:39.761774 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:17:39.761789 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:17:39.761825 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:17:39.761836 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:17:39.761911 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:39.762077 (+ 166us) server_negotiation.cc:299] Negotiation successful
0925 15:17:39.762146 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 15:17:39.766515 29961 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56800:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:17:39.773222 29961 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:17:39.795258 30421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:39.785261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55528 (local address 127.12.8.1:36335)
0925 15:17:39.787219 (+ 1958us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:39.787228 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:39.787260 (+ 32us) server_negotiation.cc:407] Connection header received
0925 15:17:39.787335 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:39.787340 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:39.787594 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:39.787780 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:17:39.788447 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:39.790579 (+ 2132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:39.792668 (+ 2089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:39.794052 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:39.794192 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:39.794468 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:39.794751 (+ 283us) server_negotiation.cc:299] Negotiation successful
0925 15:17:39.794918 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1809,"thread_start_us":1713,"threads_started":1}
I20260925 15:17:39.795297 30422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:39.785061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.1:36335 (local address 127.0.0.1:55528)
0925 15:17:39.786360 (+ 1299us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:39.786408 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 15:17:39.786616 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:17:39.787785 (+ 1169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:17:39.787792 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:17:39.787829 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:17:39.788212 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:39.788225 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:39.790747 (+ 2522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:39.790753 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:39.792487 (+ 1734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:39.792499 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:39.794270 (+ 1771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:39.794276 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:39.794530 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:39.794572 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 15:17:39.794707 (+ 135us) client_negotiation.cc:240] Negotiation successful
0925 15:17:39.794891 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":966,"mutex_wait_us":34,"thread_start_us":3429,"threads_started":1}
I20260925 15:17:39.798372 30420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:39.785457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.2:37955 (local address 127.0.0.1:40006)
0925 15:17:39.788794 (+ 3337us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:39.788815 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 15:17:39.788940 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:17:39.789478 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:17:39.789485 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:17:39.789515 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:17:39.789927 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:39.789945 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:39.792781 (+ 2836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:39.792789 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:39.795681 (+ 2892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:39.795698 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:39.797504 (+ 1806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:39.797511 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:39.797712 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:39.797746 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 15:17:39.797811 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 15:17:39.797964 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3177,"mutex_wait_us":1,"thread_start_us":995,"threads_started":1}
I20260925 15:17:39.799405 30423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:39.786054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40006 (local address 127.12.8.2:37955)
0925 15:17:39.788404 (+ 2350us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:39.788412 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:39.788918 (+ 506us) server_negotiation.cc:407] Connection header received
0925 15:17:39.789071 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:39.789078 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:39.789285 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:39.789463 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:17:39.790143 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:39.792553 (+ 2410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:39.795890 (+ 3337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:39.797266 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:39.798600 (+ 1334us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:39.798832 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:39.798909 (+ 77us) server_negotiation.cc:299] Negotiation successful
0925 15:17:39.799062 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2163,"thread_start_us":2039,"threads_started":1}
I20260925 15:17:39.799561 30082 tablet_service.cc:1467] Processing CreateTablet for tablet 625b307ac0124144b5aa2688f2a5170c (DEFAULT_TABLE table=test-table [id=23311e870d2741f190034fc3714a666b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:17:39.800995 30082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 625b307ac0124144b5aa2688f2a5170c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:17:39.802989 30213 tablet_service.cc:1467] Processing CreateTablet for tablet 625b307ac0124144b5aa2688f2a5170c (DEFAULT_TABLE table=test-table [id=23311e870d2741f190034fc3714a666b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:17:39.803983 30425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:39.789635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.3:35451 (local address 127.0.0.1:40646)
0925 15:17:39.790982 (+ 1347us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:39.791000 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 15:17:39.791107 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:17:39.796882 (+ 5775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:17:39.796887 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:17:39.796913 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:17:39.797485 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:39.797496 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:39.799442 (+ 1946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:39.799448 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:39.801075 (+ 1627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:39.801090 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:39.803215 (+ 2125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:39.803221 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:39.803424 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:39.803459 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 15:17:39.803550 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 15:17:39.803681 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1202,"thread_start_us":760,"threads_started":1}
I20260925 15:17:39.804340 30213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 625b307ac0124144b5aa2688f2a5170c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:17:39.804843 30424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:39.789248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40646 (local address 127.12.8.3:35451)
0925 15:17:39.796342 (+ 7094us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:39.796347 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:39.796374 (+ 27us) server_negotiation.cc:407] Connection header received
0925 15:17:39.796446 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:39.796451 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:39.796673 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:39.796881 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:17:39.797712 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:39.799253 (+ 1541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:39.801295 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:39.802705 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:39.802832 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:39.803094 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:39.804309 (+ 1215us) server_negotiation.cc:299] Negotiation successful
0925 15:17:39.804495 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6943,"thread_start_us":6798,"threads_started":1}
I20260925 15:17:39.808354 30343 tablet_service.cc:1467] Processing CreateTablet for tablet 625b307ac0124144b5aa2688f2a5170c (DEFAULT_TABLE table=test-table [id=23311e870d2741f190034fc3714a666b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:17:39.810214 30343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 625b307ac0124144b5aa2688f2a5170c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:17:39.812507 30426 tablet_bootstrap.cc:492] T 625b307ac0124144b5aa2688f2a5170c P f5c0ef93dbcb4dc99035de8ec45b0f9b: Bootstrap starting.
I20260925 15:17:39.816500 30426 tablet_bootstrap.cc:654] T 625b307ac0124144b5aa2688f2a5170c P f5c0ef93dbcb4dc99035de8ec45b0f9b: Neither blocks nor log segments found. Creating new log.
I20260925 15:17:39.817725 30426 log.cc:824] T 625b307ac0124144b5aa2688f2a5170c P f5c0ef93dbcb4dc99035de8ec45b0f9b: Log is configured to *not* fsync() on all Append() calls
I20260925 15:17:39.818925 30427 tablet_bootstrap.cc:492] T 625b307ac0124144b5aa2688f2a5170c P bbe8e3a6eaf54ceb8f34d33a371a669d: Bootstrap starting.
I20260925 15:17:39.823621 30427 tablet_bootstrap.cc:654] T 625b307ac0124144b5aa2688f2a5170c P bbe8e3a6eaf54ceb8f34d33a371a669d: Neither blocks nor log segments found. Creating new log.
I20260925 15:17:39.824949 30427 log.cc:824] T 625b307ac0124144b5aa2688f2a5170c P bbe8e3a6eaf54ceb8f34d33a371a669d: Log is configured to *not* fsync() on all Append() calls
I20260925 15:17:39.825116 30426 tablet_bootstrap.cc:492] T 625b307ac0124144b5aa2688f2a5170c P f5c0ef93dbcb4dc99035de8ec45b0f9b: No bootstrap required, opened a new log
I20260925 15:17:39.825393 30426 ts_tablet_manager.cc:1397] T 625b307ac0124144b5aa2688f2a5170c P f5c0ef93dbcb4dc99035de8ec45b0f9b: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.003s
I20260925 15:17:39.826823 30429 tablet_bootstrap.cc:492] T 625b307ac0124144b5aa2688f2a5170c P a69933091d2747cfbb24de6e68395b21: Bootstrap starting.
I20260925 15:17:39.827907 30427 tablet_bootstrap.cc:492] T 625b307ac0124144b5aa2688f2a5170c P bbe8e3a6eaf54ceb8f34d33a371a669d: No bootstrap required, opened a new log
I20260925 15:17:39.828215 30427 ts_tablet_manager.cc:1397] T 625b307ac0124144b5aa2688f2a5170c P bbe8e3a6eaf54ceb8f34d33a371a669d: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260925 15:17:39.830365 30429 tablet_bootstrap.cc:654] T 625b307ac0124144b5aa2688f2a5170c P a69933091d2747cfbb24de6e68395b21: Neither blocks nor log segments found. Creating new log.
I20260925 15:17:39.831717 30429 log.cc:824] T 625b307ac0124144b5aa2688f2a5170c P a69933091d2747cfbb24de6e68395b21: Log is configured to *not* fsync() on all Append() calls
I20260925 15:17:39.837339 30426 raft_consensus.cc:348] T 625b307ac0124144b5aa2688f2a5170c P f5c0ef93dbcb4dc99035de8ec45b0f9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5c0ef93dbcb4dc99035de8ec45b0f9b" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 36335 } } peers { permanent_uuid: "bbe8e3a6eaf54ceb8f34d33a371a669d" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 37955 } } peers { permanent_uuid: "a69933091d2747cfbb24de6e68395b21" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 35451 } }
I20260925 15:17:39.837921 30426 raft_consensus.cc:374] T 625b307ac0124144b5aa2688f2a5170c P f5c0ef93dbcb4dc99035de8ec45b0f9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:17:39.838070 30426 raft_consensus.cc:729] T 625b307ac0124144b5aa2688f2a5170c P f5c0ef93dbcb4dc99035de8ec45b0f9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5c0ef93dbcb4dc99035de8ec45b0f9b, State: Initialized, Role: FOLLOWER
I20260925 15:17:39.838616 30427 raft_consensus.cc:348] T 625b307ac0124144b5aa2688f2a5170c P bbe8e3a6eaf54ceb8f34d33a371a669d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5c0ef93dbcb4dc99035de8ec45b0f9b" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 36335 } } peers { permanent_uuid: "bbe8e3a6eaf54ceb8f34d33a371a669d" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 37955 } } peers { permanent_uuid: "a69933091d2747cfbb24de6e68395b21" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 35451 } }
I20260925 15:17:39.839095 30427 raft_consensus.cc:374] T 625b307ac0124144b5aa2688f2a5170c P bbe8e3a6eaf54ceb8f34d33a371a669d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:17:39.838932 30426 consensus_queue.cc:260] T 625b307ac0124144b5aa2688f2a5170c P f5c0ef93dbcb4dc99035de8ec45b0f9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5c0ef93dbcb4dc99035de8ec45b0f9b" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 36335 } } peers { permanent_uuid: "bbe8e3a6eaf54ceb8f34d33a371a669d" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 37955 } } peers { permanent_uuid: "a69933091d2747cfbb24de6e68395b21" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 35451 } }
I20260925 15:17:39.839229 30427 raft_consensus.cc:729] T 625b307ac0124144b5aa2688f2a5170c P bbe8e3a6eaf54ceb8f34d33a371a669d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbe8e3a6eaf54ceb8f34d33a371a669d, State: Initialized, Role: FOLLOWER
I20260925 15:17:39.840390 30147 heartbeater.cc:503] Master 127.12.8.62:40305 was elected leader, sending a full tablet report...
I20260925 15:17:39.841568 30426 ts_tablet_manager.cc:1428] T 625b307ac0124144b5aa2688f2a5170c P f5c0ef93dbcb4dc99035de8ec45b0f9b: Time spent starting tablet: real 0.016s user 0.014s sys 0.001s
I20260925 15:17:39.839954 30427 consensus_queue.cc:260] T 625b307ac0124144b5aa2688f2a5170c P bbe8e3a6eaf54ceb8f34d33a371a669d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5c0ef93dbcb4dc99035de8ec45b0f9b" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 36335 } } peers { permanent_uuid: "bbe8e3a6eaf54ceb8f34d33a371a669d" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 37955 } } peers { permanent_uuid: "a69933091d2747cfbb24de6e68395b21" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 35451 } }
I20260925 15:17:39.842355 30429 tablet_bootstrap.cc:492] T 625b307ac0124144b5aa2688f2a5170c P a69933091d2747cfbb24de6e68395b21: No bootstrap required, opened a new log
I20260925 15:17:39.842626 30429 ts_tablet_manager.cc:1397] T 625b307ac0124144b5aa2688f2a5170c P a69933091d2747cfbb24de6e68395b21: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.009s
I20260925 15:17:39.849651 30278 heartbeater.cc:503] Master 127.12.8.62:40305 was elected leader, sending a full tablet report...
I20260925 15:17:39.850399 30427 ts_tablet_manager.cc:1428] T 625b307ac0124144b5aa2688f2a5170c P bbe8e3a6eaf54ceb8f34d33a371a669d: Time spent starting tablet: real 0.022s user 0.013s sys 0.006s
I20260925 15:17:39.853401 30429 raft_consensus.cc:348] T 625b307ac0124144b5aa2688f2a5170c P a69933091d2747cfbb24de6e68395b21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5c0ef93dbcb4dc99035de8ec45b0f9b" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 36335 } } peers { permanent_uuid: "bbe8e3a6eaf54ceb8f34d33a371a669d" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 37955 } } peers { permanent_uuid: "a69933091d2747cfbb24de6e68395b21" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 35451 } }
I20260925 15:17:39.853771 30429 raft_consensus.cc:374] T 625b307ac0124144b5aa2688f2a5170c P a69933091d2747cfbb24de6e68395b21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:17:39.853947 30429 raft_consensus.cc:729] T 625b307ac0124144b5aa2688f2a5170c P a69933091d2747cfbb24de6e68395b21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a69933091d2747cfbb24de6e68395b21, State: Initialized, Role: FOLLOWER
I20260925 15:17:39.854573 30429 consensus_queue.cc:260] T 625b307ac0124144b5aa2688f2a5170c P a69933091d2747cfbb24de6e68395b21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5c0ef93dbcb4dc99035de8ec45b0f9b" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 36335 } } peers { permanent_uuid: "bbe8e3a6eaf54ceb8f34d33a371a669d" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 37955 } } peers { permanent_uuid: "a69933091d2747cfbb24de6e68395b21" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 35451 } }
I20260925 15:17:39.855652 30408 heartbeater.cc:503] Master 127.12.8.62:40305 was elected leader, sending a full tablet report...
I20260925 15:17:39.856343 30429 ts_tablet_manager.cc:1428] T 625b307ac0124144b5aa2688f2a5170c P a69933091d2747cfbb24de6e68395b21: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260925 15:17:39.946125 30434 raft_consensus.cc:482] T 625b307ac0124144b5aa2688f2a5170c P a69933091d2747cfbb24de6e68395b21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:17:39.946476 30434 raft_consensus.cc:504] T 625b307ac0124144b5aa2688f2a5170c P a69933091d2747cfbb24de6e68395b21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5c0ef93dbcb4dc99035de8ec45b0f9b" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 36335 } } peers { permanent_uuid: "bbe8e3a6eaf54ceb8f34d33a371a669d" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 37955 } } peers { permanent_uuid: "a69933091d2747cfbb24de6e68395b21" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 35451 } }
I20260925 15:17:39.948148 30434 leader_election.cc:288] T 625b307ac0124144b5aa2688f2a5170c P a69933091d2747cfbb24de6e68395b21 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f5c0ef93dbcb4dc99035de8ec45b0f9b (127.12.8.1:36335), bbe8e3a6eaf54ceb8f34d33a371a669d (127.12.8.2:37955)
I20260925 15:17:39.955008 30421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:39.948623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.3:50113 (local address 127.12.8.1:36335)
0925 15:17:39.948857 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:39.948866 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:39.948892 (+ 26us) server_negotiation.cc:407] Connection header received
0925 15:17:39.948981 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:39.948989 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:39.949081 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:39.949208 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:17:39.949679 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:39.950666 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:39.952941 (+ 2275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:39.954059 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:39.954221 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:39.954328 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:39.954609 (+ 281us) server_negotiation.cc:299] Negotiation successful
0925 15:17:39.954735 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 15:17:39.955106 30410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:39.948443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.1:36335 (local address 127.12.8.3:50113)
0925 15:17:39.948640 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:39.948663 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 15:17:39.948790 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:17:39.949272 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:17:39.949279 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:17:39.949314 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:17:39.949542 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:39.949551 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:39.950790 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:39.950794 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:39.952768 (+ 1974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:39.952783 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:39.954275 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:39.954280 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:39.954437 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:39.954465 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 15:17:39.954554 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 15:17:39.954727 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260925 15:17:39.955619 30423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:39.948569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.3:37917 (local address 127.12.8.2:37955)
0925 15:17:39.948744 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:39.948750 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:39.949367 (+ 617us) server_negotiation.cc:407] Connection header received
0925 15:17:39.949511 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:39.949515 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:39.949584 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:39.949711 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:17:39.950236 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:39.951074 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:39.953311 (+ 2237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:39.954507 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:39.954650 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:39.954755 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:39.955266 (+ 511us) server_negotiation.cc:299] Negotiation successful
0925 15:17:39.955435 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 15:17:39.956110 30435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:39.948443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.2:37955 (local address 127.12.8.3:37917)
0925 15:17:39.949251 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:39.949267 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:17:39.949390 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:17:39.949712 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:17:39.949715 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:17:39.949733 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:17:39.950042 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:39.950055 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:39.951252 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:39.951258 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:39.953151 (+ 1893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:39.953160 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:39.954704 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:39.954710 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:39.954931 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:39.954974 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 15:17:39.955729 (+ 755us) client_negotiation.cc:240] Negotiation successful
0925 15:17:39.955869 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":659,"mutex_wait_us":148,"thread_start_us":306,"threads_started":1}
I20260925 15:17:39.956235 30102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "625b307ac0124144b5aa2688f2a5170c" candidate_uuid: "a69933091d2747cfbb24de6e68395b21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5c0ef93dbcb4dc99035de8ec45b0f9b" is_pre_election: true
I20260925 15:17:39.956818 30102 raft_consensus.cc:2454] T 625b307ac0124144b5aa2688f2a5170c P f5c0ef93dbcb4dc99035de8ec45b0f9b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a69933091d2747cfbb24de6e68395b21 in term 0.
I20260925 15:17:39.957021 30233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "625b307ac0124144b5aa2688f2a5170c" candidate_uuid: "a69933091d2747cfbb24de6e68395b21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe8e3a6eaf54ceb8f34d33a371a669d" is_pre_election: true
I20260925 15:17:39.957499 30296 leader_election.cc:302] T 625b307ac0124144b5aa2688f2a5170c P a69933091d2747cfbb24de6e68395b21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a69933091d2747cfbb24de6e68395b21, f5c0ef93dbcb4dc99035de8ec45b0f9b; no voters:
I20260925 15:17:39.957544 30233 raft_consensus.cc:2454] T 625b307ac0124144b5aa2688f2a5170c P bbe8e3a6eaf54ceb8f34d33a371a669d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a69933091d2747cfbb24de6e68395b21 in term 0.
I20260925 15:17:39.957904 30434 raft_consensus.cc:2781] T 625b307ac0124144b5aa2688f2a5170c P a69933091d2747cfbb24de6e68395b21 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:17:39.958060 30434 raft_consensus.cc:482] T 625b307ac0124144b5aa2688f2a5170c P a69933091d2747cfbb24de6e68395b21 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:17:39.958246 30434 raft_consensus.cc:3037] T 625b307ac0124144b5aa2688f2a5170c P a69933091d2747cfbb24de6e68395b21 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:17:39.960615 30434 raft_consensus.cc:504] T 625b307ac0124144b5aa2688f2a5170c P a69933091d2747cfbb24de6e68395b21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5c0ef93dbcb4dc99035de8ec45b0f9b" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 36335 } } peers { permanent_uuid: "bbe8e3a6eaf54ceb8f34d33a371a669d" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 37955 } } peers { permanent_uuid: "a69933091d2747cfbb24de6e68395b21" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 35451 } }
I20260925 15:17:39.961290 30434 leader_election.cc:288] T 625b307ac0124144b5aa2688f2a5170c P a69933091d2747cfbb24de6e68395b21 [CANDIDATE]: Term 1 election: Requested vote from peers f5c0ef93dbcb4dc99035de8ec45b0f9b (127.12.8.1:36335), bbe8e3a6eaf54ceb8f34d33a371a669d (127.12.8.2:37955)
I20260925 15:17:39.961644 30102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "625b307ac0124144b5aa2688f2a5170c" candidate_uuid: "a69933091d2747cfbb24de6e68395b21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5c0ef93dbcb4dc99035de8ec45b0f9b"
I20260925 15:17:39.961851 30102 raft_consensus.cc:3037] T 625b307ac0124144b5aa2688f2a5170c P f5c0ef93dbcb4dc99035de8ec45b0f9b [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:17:39.961835 30233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "625b307ac0124144b5aa2688f2a5170c" candidate_uuid: "a69933091d2747cfbb24de6e68395b21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe8e3a6eaf54ceb8f34d33a371a669d"
I20260925 15:17:39.962076 30233 raft_consensus.cc:3037] T 625b307ac0124144b5aa2688f2a5170c P bbe8e3a6eaf54ceb8f34d33a371a669d [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:17:39.964044 30102 raft_consensus.cc:2454] T 625b307ac0124144b5aa2688f2a5170c P f5c0ef93dbcb4dc99035de8ec45b0f9b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a69933091d2747cfbb24de6e68395b21 in term 1.
I20260925 15:17:39.964404 30296 leader_election.cc:302] T 625b307ac0124144b5aa2688f2a5170c P a69933091d2747cfbb24de6e68395b21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a69933091d2747cfbb24de6e68395b21, f5c0ef93dbcb4dc99035de8ec45b0f9b; no voters:
I20260925 15:17:39.964617 30233 raft_consensus.cc:2454] T 625b307ac0124144b5aa2688f2a5170c P bbe8e3a6eaf54ceb8f34d33a371a669d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a69933091d2747cfbb24de6e68395b21 in term 1.
I20260925 15:17:39.964687 30434 raft_consensus.cc:2781] T 625b307ac0124144b5aa2688f2a5170c P a69933091d2747cfbb24de6e68395b21 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:17:39.965200 30434 raft_consensus.cc:686] T 625b307ac0124144b5aa2688f2a5170c P a69933091d2747cfbb24de6e68395b21 [term 1 LEADER]: Becoming Leader. State: Replica: a69933091d2747cfbb24de6e68395b21, State: Running, Role: LEADER
I20260925 15:17:39.965754 30434 consensus_queue.cc:237] T 625b307ac0124144b5aa2688f2a5170c P a69933091d2747cfbb24de6e68395b21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5c0ef93dbcb4dc99035de8ec45b0f9b" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 36335 } } peers { permanent_uuid: "bbe8e3a6eaf54ceb8f34d33a371a669d" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 37955 } } peers { permanent_uuid: "a69933091d2747cfbb24de6e68395b21" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 35451 } }
I20260925 15:17:39.970707 29961 catalog_manager.cc:5495] T 625b307ac0124144b5aa2688f2a5170c P a69933091d2747cfbb24de6e68395b21 reported cstate change: term changed from 0 to 1, leader changed from <none> to a69933091d2747cfbb24de6e68395b21 (127.12.8.3). New cstate: current_term: 1 leader_uuid: "a69933091d2747cfbb24de6e68395b21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5c0ef93dbcb4dc99035de8ec45b0f9b" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 36335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bbe8e3a6eaf54ceb8f34d33a371a669d" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 37955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a69933091d2747cfbb24de6e68395b21" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 35451 } health_report { overall_health: HEALTHY } } }
W20260925 15:17:39.978787 30409 tablet.cc:2369] T 625b307ac0124144b5aa2688f2a5170c P a69933091d2747cfbb24de6e68395b21: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:17:39.987022 30424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:39.981835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40652 (local address 127.12.8.3:35451)
0925 15:17:39.981993 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:39.981997 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:39.982014 (+ 17us) server_negotiation.cc:407] Connection header received
0925 15:17:39.982233 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:39.982243 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:39.982380 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:39.982503 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:17:39.982837 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:39.983594 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:39.984954 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:39.985405 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:39.985501 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:39.985979 (+ 478us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:17:39.986558 (+ 579us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:17:39.986671 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:39.986787 (+ 116us) server_negotiation.cc:299] Negotiation successful
0925 15:17:39.986847 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 15:17:39.995564 30102 raft_consensus.cc:1261] T 625b307ac0124144b5aa2688f2a5170c P f5c0ef93dbcb4dc99035de8ec45b0f9b [term 1 FOLLOWER]: Refusing update from remote peer a69933091d2747cfbb24de6e68395b21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:17:39.995564 30233 raft_consensus.cc:1261] T 625b307ac0124144b5aa2688f2a5170c P bbe8e3a6eaf54ceb8f34d33a371a669d [term 1 FOLLOWER]: Refusing update from remote peer a69933091d2747cfbb24de6e68395b21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:17:39.996531 30436 consensus_queue.cc:1035] T 625b307ac0124144b5aa2688f2a5170c P a69933091d2747cfbb24de6e68395b21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5c0ef93dbcb4dc99035de8ec45b0f9b" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 36335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:17:39.996908 30434 consensus_queue.cc:1035] T 625b307ac0124144b5aa2688f2a5170c P a69933091d2747cfbb24de6e68395b21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbe8e3a6eaf54ceb8f34d33a371a669d" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 37955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:17:40.007263 30439 mvcc.cc:205] Tried to move back new op lower bound from 7333271388131573760 to 7333271388024324096. Current Snapshot: MvccSnapshot[applied={T|T < 7333271388131573760}]
I20260925 15:17:40.011840 30440 mvcc.cc:205] Tried to move back new op lower bound from 7333271388131573760 to 7333271388024324096. Current Snapshot: MvccSnapshot[applied={T|T < 7333271388131573760}]
I20260925 15:17:40.023353 29961 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56800:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:17:40.023581 29961 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56800:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:17:40.026955 29961 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2f589af505144a87bf0ebabea4a3a5c4: Sending DeleteTablet for 3 replicas of tablet 625b307ac0124144b5aa2688f2a5170c
I20260925 15:17:40.028190 30082 tablet_service.cc:1514] Processing DeleteTablet for tablet 625b307ac0124144b5aa2688f2a5170c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:17:40 UTC) from {username='slave'} at 127.0.0.1:55528
I20260925 15:17:40.028439 30343 tablet_service.cc:1514] Processing DeleteTablet for tablet 625b307ac0124144b5aa2688f2a5170c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:17:40 UTC) from {username='slave'} at 127.0.0.1:40646
I20260925 15:17:40.028924 30213 tablet_service.cc:1514] Processing DeleteTablet for tablet 625b307ac0124144b5aa2688f2a5170c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:17:40 UTC) from {username='slave'} at 127.0.0.1:40006
I20260925 15:17:40.030638 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 30020
I20260925 15:17:40.030709 30449 tablet_replica.cc:330] T 625b307ac0124144b5aa2688f2a5170c P f5c0ef93dbcb4dc99035de8ec45b0f9b: stopping tablet replica
I20260925 15:17:40.031222 30449 raft_consensus.cc:2229] T 625b307ac0124144b5aa2688f2a5170c P f5c0ef93dbcb4dc99035de8ec45b0f9b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:17:40.031817 30449 raft_consensus.cc:2258] T 625b307ac0124144b5aa2688f2a5170c P f5c0ef93dbcb4dc99035de8ec45b0f9b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:17:40.034526 30449 ts_tablet_manager.cc:1905] T 625b307ac0124144b5aa2688f2a5170c P f5c0ef93dbcb4dc99035de8ec45b0f9b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:17:40.038774 30451 tablet_replica.cc:330] T 625b307ac0124144b5aa2688f2a5170c P bbe8e3a6eaf54ceb8f34d33a371a669d: stopping tablet replica
I20260925 15:17:40.039233 30451 raft_consensus.cc:2229] T 625b307ac0124144b5aa2688f2a5170c P bbe8e3a6eaf54ceb8f34d33a371a669d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:17:40.041148 30451 raft_consensus.cc:2258] T 625b307ac0124144b5aa2688f2a5170c P bbe8e3a6eaf54ceb8f34d33a371a669d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:17:40.042973 30449 ts_tablet_manager.cc:1918] T 625b307ac0124144b5aa2688f2a5170c P f5c0ef93dbcb4dc99035de8ec45b0f9b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:17:40.043146 30449 log.cc:1196] T 625b307ac0124144b5aa2688f2a5170c P f5c0ef93dbcb4dc99035de8ec45b0f9b: Deleting WAL directory at /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349162370577-12320-0/minicluster-data/ts-0/wal/wals/625b307ac0124144b5aa2688f2a5170c
I20260925 15:17:40.043708 30449 ts_tablet_manager.cc:1939] T 625b307ac0124144b5aa2688f2a5170c P f5c0ef93dbcb4dc99035de8ec45b0f9b: Deleting consensus metadata
I20260925 15:17:40.043773 30451 ts_tablet_manager.cc:1905] T 625b307ac0124144b5aa2688f2a5170c P bbe8e3a6eaf54ceb8f34d33a371a669d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:17:40.045531 29945 catalog_manager.cc:4841] TS f5c0ef93dbcb4dc99035de8ec45b0f9b (127.12.8.1:36335): tablet 625b307ac0124144b5aa2688f2a5170c (table test-table [id=23311e870d2741f190034fc3714a666b]) successfully deleted
I20260925 15:17:40.049889 30450 tablet_replica.cc:330] T 625b307ac0124144b5aa2688f2a5170c P a69933091d2747cfbb24de6e68395b21: stopping tablet replica
I20260925 15:17:40.050341 30450 raft_consensus.cc:2229] T 625b307ac0124144b5aa2688f2a5170c P a69933091d2747cfbb24de6e68395b21 [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:17:40.051097 30450 raft_consensus.cc:2258] T 625b307ac0124144b5aa2688f2a5170c P a69933091d2747cfbb24de6e68395b21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:17:40.051617 30451 ts_tablet_manager.cc:1918] T 625b307ac0124144b5aa2688f2a5170c P bbe8e3a6eaf54ceb8f34d33a371a669d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:17:40.051764 30451 log.cc:1196] T 625b307ac0124144b5aa2688f2a5170c P bbe8e3a6eaf54ceb8f34d33a371a669d: Deleting WAL directory at /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349162370577-12320-0/minicluster-data/ts-1/wal/wals/625b307ac0124144b5aa2688f2a5170c
I20260925 15:17:40.052369 30451 ts_tablet_manager.cc:1939] T 625b307ac0124144b5aa2688f2a5170c P bbe8e3a6eaf54ceb8f34d33a371a669d: Deleting consensus metadata
I20260925 15:17:40.053380 30450 ts_tablet_manager.cc:1905] T 625b307ac0124144b5aa2688f2a5170c P a69933091d2747cfbb24de6e68395b21: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:17:40.053860 30421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:40.033279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55544 (local address 127.12.8.1:36335)
0925 15:17:40.033446 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:40.033452 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:40.044833 (+ 11381us) server_negotiation.cc:407] Connection header received
0925 15:17:40.045329 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:40.045335 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:40.045456 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:40.045535 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:17:40.046274 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:40.047382 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:40.049358 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:40.049990 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:40.050085 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:40.052003 (+ 1918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:17:40.052013 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:17:40.052029 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:17:40.052851 (+ 822us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:17:40.052909 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:17:40.052943 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:17:40.053041 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:40.053382 (+ 341us) server_negotiation.cc:299] Negotiation successful
0925 15:17:40.053506 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 15:17:40.054257 29946 catalog_manager.cc:4841] TS bbe8e3a6eaf54ceb8f34d33a371a669d (127.12.8.2:37955): tablet 625b307ac0124144b5aa2688f2a5170c (table test-table [id=23311e870d2741f190034fc3714a666b]) successfully deleted
I20260925 15:17:40.055270 30142 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:17:40.060896 30450 ts_tablet_manager.cc:1918] T 625b307ac0124144b5aa2688f2a5170c P a69933091d2747cfbb24de6e68395b21: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:17:40.061054 30450 log.cc:1196] T 625b307ac0124144b5aa2688f2a5170c P a69933091d2747cfbb24de6e68395b21: Deleting WAL directory at /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349162370577-12320-0/minicluster-data/ts-2/wal/wals/625b307ac0124144b5aa2688f2a5170c
I20260925 15:17:40.061592 30450 ts_tablet_manager.cc:1939] T 625b307ac0124144b5aa2688f2a5170c P a69933091d2747cfbb24de6e68395b21: Deleting consensus metadata
I20260925 15:17:40.063377 29945 catalog_manager.cc:4841] TS a69933091d2747cfbb24de6e68395b21 (127.12.8.3:35451): tablet 625b307ac0124144b5aa2688f2a5170c (table test-table [id=23311e870d2741f190034fc3714a666b]) successfully deleted
I20260925 15:17:40.145143 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 30020
I20260925 15:17:40.163555 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 30151
I20260925 15:17:40.170500 30423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:40.164379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40020 (local address 127.12.8.2:37955)
0925 15:17:40.164558 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:40.164567 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:40.164588 (+ 21us) server_negotiation.cc:407] Connection header received
0925 15:17:40.164799 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:40.164805 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:40.164922 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:40.165045 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:17:40.165433 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:40.166167 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:40.167481 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:40.167896 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:40.168025 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:40.169118 (+ 1093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:17:40.169134 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:17:40.169148 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:17:40.169710 (+ 562us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:17:40.169756 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:17:40.169778 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:17:40.169910 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:40.170041 (+ 131us) server_negotiation.cc:299] Negotiation successful
0925 15:17:40.170146 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 15:17:40.171187 30273 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:17:40.260654 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 30151
I20260925 15:17:40.278396 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 30281
I20260925 15:17:40.286381 30424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:40.279224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40658 (local address 127.12.8.3:35451)
0925 15:17:40.279456 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:40.279462 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:40.279484 (+ 22us) server_negotiation.cc:407] Connection header received
0925 15:17:40.279593 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:40.279598 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:40.279720 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:40.279841 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:17:40.280265 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:40.281494 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:40.282898 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:40.283594 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:40.283732 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:40.284736 (+ 1004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:17:40.284748 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:17:40.284768 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:17:40.285554 (+ 786us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:17:40.285617 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:17:40.285648 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:17:40.285764 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:40.285867 (+ 103us) server_negotiation.cc:299] Negotiation successful
0925 15:17:40.285958 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260925 15:17:40.287056 30403 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:17:40.409186 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 30281
I20260925 15:17:40.428752 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 29930
I20260925 15:17:40.429898 29991 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:17:40.537118 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 29930
2026-09-25T15:17:40Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2745 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349162370577-12320-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:17:40 dist-test-slave-p468 krb5kdc[30459](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:17:40 dist-test-slave-p468 krb5kdc[30459](info): set up 2 sockets
Sep 25 15:17:40 dist-test-slave-p468 krb5kdc[30459](info): commencing operation
krb5kdc: starting...
W20260925 15:17:41.634441 12320 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.005s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349162370577-12320-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349162370577-12320-0/krb5kdc/test-user.keytab.
Sep 25 15:17:41 dist-test-slave-p468 krb5kdc[30459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349461, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:17:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:17:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.8.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab.
Entry for principal kudu/127.12.8.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab.
I20260925 15:17:41.877898 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349162370577-12320-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349162370577-12320-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349162370577-12320-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.62:46883
--webserver_interface=127.12.8.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab
--principal=kudu/127.12.8.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:38919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.62:46883
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:17:42.198555 30472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:17:42.198910 30472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:17:42.199074 30472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:17:42.206086 30472 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:17:42.206177 30472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:17:42.206245 30472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:17:42.206348 30472 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:17:42.206398 30472 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:17:42.215951 30472 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:38919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349162370577-12320-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349162370577-12320-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.8.62:46883
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab
--principal=kudu/127.12.8.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.62:46883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.8.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349162370577-12320-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:17:42.217149 30472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:17:42.218465 30472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:17:42.230620 30477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:17:42.230527 30480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:17:42.230561 30478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:17:42.231243 30472 server_base.cc:1034] running on GCE node
I20260925 15:17:42.232076 30472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:17:42.233415 30472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:17:42.234650 30472 hybrid_clock.cc:648] HybridClock initialized: now 1790349462234614 us; error 58 us; skew 500 ppm
I20260925 15:17:42.234984 30472 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:17:42 dist-test-slave-p468 krb5kdc[30459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349462, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:17:42.238031 30472 init.cc:377] Logged in from keytab as kudu/127.12.8.62@KRBTEST.COM (short username kudu)
I20260925 15:17:42.239596 30472 webserver.cc:460] Webserver started at http://127.12.8.62:40679/ using document root <none> and password file <none>
I20260925 15:17:42.240253 30472 fs_manager.cc:359] Metadata directory not provided
I20260925 15:17:42.240408 30472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:17:42.240722 30472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:17:42.243676 30472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349162370577-12320-0/minicluster-data/master-0/data/instance:
uuid: "3e75f552d2574b20b5446e426140cd6b"
format_stamp: "Formatted at 2026-09-25 15:17:42 on dist-test-slave-p468"
server_key: "cc5af0b660c4822c263631d2705b227e"
server_key_iv: "307f0607a6a22dd36b2e8ae1c11b1c71"
server_key_version: "encryptionkey@0"
I20260925 15:17:42.244423 30472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349162370577-12320-0/minicluster-data/master-0/wal/instance:
uuid: "3e75f552d2574b20b5446e426140cd6b"
format_stamp: "Formatted at 2026-09-25 15:17:42 on dist-test-slave-p468"
server_key: "cc5af0b660c4822c263631d2705b227e"
server_key_iv: "307f0607a6a22dd36b2e8ae1c11b1c71"
server_key_version: "encryptionkey@0"
I20260925 15:17:42.249357 30472 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260925 15:17:42.252780 30487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:17:42.254141 30472 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 15:17:42.254359 30472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349162370577-12320-0/minicluster-data/master-0/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349162370577-12320-0/minicluster-data/master-0/wal
uuid: "3e75f552d2574b20b5446e426140cd6b"
format_stamp: "Formatted at 2026-09-25 15:17:42 on dist-test-slave-p468"
server_key: "cc5af0b660c4822c263631d2705b227e"
server_key_iv: "307f0607a6a22dd36b2e8ae1c11b1c71"
server_key_version: "encryptionkey@0"
I20260925 15:17:42.254529 30472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349162370577-12320-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349162370577-12320-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349162370577-12320-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:17:42.267256 30472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:17:42.271396 30472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:17:42.271636 30472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:17:42.289070 30472 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.62:46883
I20260925 15:17:42.289099 30538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.62:46883 every 8 connection(s)
I20260925 15:17:42.290591 30472 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
I20260925 15:17:42.294360 30539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:17:42.300839 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 30472
I20260925 15:17:42.301044 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349162370577-12320-0/minicluster-data/master-0/wal/instance
I20260925 15:17:42.301599 12320 external_mini_cluster.cc:1428] Setting key e670da9c4aeea8060c1c1bf85a710854
I20260925 15:17:42.305325 30539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e75f552d2574b20b5446e426140cd6b: Bootstrap starting.
I20260925 15:17:42.309036 30539 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e75f552d2574b20b5446e426140cd6b: Neither blocks nor log segments found. Creating new log.
I20260925 15:17:42.310052 30539 log.cc:824] T 00000000000000000000000000000000 P 3e75f552d2574b20b5446e426140cd6b: Log is configured to *not* fsync() on all Append() calls
I20260925 15:17:42.312928 30539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e75f552d2574b20b5446e426140cd6b: No bootstrap required, opened a new log
Sep 25 15:17:42 dist-test-slave-p468 krb5kdc[30459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349461, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
I20260925 15:17:42.323632 30539 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e75f552d2574b20b5446e426140cd6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e75f552d2574b20b5446e426140cd6b" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 46883 } }
I20260925 15:17:42.324087 30539 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e75f552d2574b20b5446e426140cd6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:17:42.324204 30539 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e75f552d2574b20b5446e426140cd6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e75f552d2574b20b5446e426140cd6b, State: Initialized, Role: FOLLOWER
I20260925 15:17:42.324951 30539 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e75f552d2574b20b5446e426140cd6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e75f552d2574b20b5446e426140cd6b" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 46883 } }
I20260925 15:17:42.325235 30539 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e75f552d2574b20b5446e426140cd6b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:17:42.325379 30539 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e75f552d2574b20b5446e426140cd6b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:17:42.325558 30539 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e75f552d2574b20b5446e426140cd6b [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:17:42.326649 30542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:42.303609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36788 (local address 127.12.8.62:46883)
0925 15:17:42.304388 (+ 779us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:42.304400 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:42.304439 (+ 39us) server_negotiation.cc:407] Connection header received
0925 15:17:42.306296 (+ 1857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:42.306318 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:42.306688 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:42.307024 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:17:42.308949 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:42.310416 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:42.312178 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:42.312835 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:42.312965 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:42.317699 (+ 4734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:17:42.317711 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:17:42.317735 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:17:42.317785 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:17:42.321903 (+ 4118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:17:42.322656 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:17:42.322664 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:17:42.322672 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:17:42.322886 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:17:42.323248 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:17:42.323253 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:17:42.323256 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:17:42.323956 (+ 700us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:17:42.324246 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:42.324741 (+ 495us) server_negotiation.cc:299] Negotiation successful
0925 15:17:42.325182 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":385,"threads_started":1}
I20260925 15:17:42.328748 30539 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e75f552d2574b20b5446e426140cd6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e75f552d2574b20b5446e426140cd6b" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 46883 } }
I20260925 15:17:42.329291 30539 leader_election.cc:302] T 00000000000000000000000000000000 P 3e75f552d2574b20b5446e426140cd6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e75f552d2574b20b5446e426140cd6b; no voters:
I20260925 15:17:42.329972 30539 leader_election.cc:288] T 00000000000000000000000000000000 P 3e75f552d2574b20b5446e426140cd6b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:17:42.330104 30544 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e75f552d2574b20b5446e426140cd6b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:17:42.330886 30544 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e75f552d2574b20b5446e426140cd6b [term 1 LEADER]: Becoming Leader. State: Replica: 3e75f552d2574b20b5446e426140cd6b, State: Running, Role: LEADER
I20260925 15:17:42.331447 30544 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e75f552d2574b20b5446e426140cd6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e75f552d2574b20b5446e426140cd6b" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 46883 } }
I20260925 15:17:42.332084 30539 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e75f552d2574b20b5446e426140cd6b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:17:42.334554 30545 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e75f552d2574b20b5446e426140cd6b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e75f552d2574b20b5446e426140cd6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e75f552d2574b20b5446e426140cd6b" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 46883 } } }
I20260925 15:17:42.334988 30545 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e75f552d2574b20b5446e426140cd6b [sys.catalog]: This master's current role is: LEADER
I20260925 15:17:42.338580 30546 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e75f552d2574b20b5446e426140cd6b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e75f552d2574b20b5446e426140cd6b. Latest consensus state: current_term: 1 leader_uuid: "3e75f552d2574b20b5446e426140cd6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e75f552d2574b20b5446e426140cd6b" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 46883 } } }
I20260925 15:17:42.338867 30546 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e75f552d2574b20b5446e426140cd6b [sys.catalog]: This master's current role is: LEADER
I20260925 15:17:42.339399 30552 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:17:42.344064 30552 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:17:42.352782 30552 catalog_manager.cc:1347] Generated new cluster ID: 62cbeb0229c64ae58508952ff2bc8997
I20260925 15:17:42.352887 30552 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:17:42.368921 30552 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:17:42.370764 30552 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:17:42.380156 30552 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3e75f552d2574b20b5446e426140cd6b: Generated new TSK 0
I20260925 15:17:42.381160 30552 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.8.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.1.keytab.
Entry for principal kudu/127.12.8.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.1.keytab.
I20260925 15:17:42.446015 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349162370577-12320-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349162370577-12320-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349162370577-12320-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349162370577-12320-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.1:0
--local_ip_for_outbound_sockets=127.12.8.1
--webserver_interface=127.12.8.1
--webserver_port=0
--tserver_master_addrs=127.12.8.62:46883
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.1.keytab
--principal=kudu/127.12.8.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:38919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:17:42.773087 30565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:17:42.773368 30565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:17:42.773561 30565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:17:42.780998 30565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:17:42.781105 30565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:17:42.781328 30565 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.1
I20260925 15:17:42.791026 30565 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:38919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349162370577-12320-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349162370577-12320-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.1.keytab
--principal=kudu/127.12.8.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349162370577-12320-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.8.1
--webserver_port=0
--tserver_master_addrs=127.12.8.62:46883
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.1
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349162370577-12320-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:17:42.792344 30565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:17:42.793687 30565 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:17:42.807485 30573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:17:42.807520 30570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:17:42.807363 30571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:17:42.808158 30565 server_base.cc:1034] running on GCE node
I20260925 15:17:42.808737 30565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:17:42.809592 30565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:17:42.810788 30565 hybrid_clock.cc:648] HybridClock initialized: now 1790349462810731 us; error 53 us; skew 500 ppm
I20260925 15:17:42.811055 30565 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:17:42 dist-test-slave-p468 krb5kdc[30459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349462, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:17:42.814257 30565 init.cc:377] Logged in from keytab as kudu/127.12.8.1@KRBTEST.COM (short username kudu)
I20260925 15:17:42.815724 30565 webserver.cc:460] Webserver started at http://127.12.8.1:42659/ using document root <none> and password file <none>
I20260925 15:17:42.816411 30565 fs_manager.cc:359] Metadata directory not provided
I20260925 15:17:42.816553 30565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:17:42.816890 30565 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:17:42.819952 30565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349162370577-12320-0/minicluster-data/ts-0/data/instance:
uuid: "196922ce8c204035b9a27a502b2ffba7"
format_stamp: "Formatted at 2026-09-25 15:17:42 on dist-test-slave-p468"
server_key: "7c459cb1d71b9c95e2be4753da993f03"
server_key_iv: "15306ecc69a755a92ac3183847051d58"
server_key_version: "encryptionkey@0"
I20260925 15:17:42.820767 30565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349162370577-12320-0/minicluster-data/ts-0/wal/instance:
uuid: "196922ce8c204035b9a27a502b2ffba7"
format_stamp: "Formatted at 2026-09-25 15:17:42 on dist-test-slave-p468"
server_key: "7c459cb1d71b9c95e2be4753da993f03"
server_key_iv: "15306ecc69a755a92ac3183847051d58"
server_key_version: "encryptionkey@0"
I20260925 15:17:42.826339 30565 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 15:17:42.829912 30580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:17:42.831447 30565 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:17:42.831655 30565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349162370577-12320-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349162370577-12320-0/minicluster-data/ts-0/wal
uuid: "196922ce8c204035b9a27a502b2ffba7"
format_stamp: "Formatted at 2026-09-25 15:17:42 on dist-test-slave-p468"
server_key: "7c459cb1d71b9c95e2be4753da993f03"
server_key_iv: "15306ecc69a755a92ac3183847051d58"
server_key_version: "encryptionkey@0"
I20260925 15:17:42.831892 30565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349162370577-12320-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349162370577-12320-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349162370577-12320-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:17:42.844055 30565 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:17:42.847887 30565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:17:42.848184 30565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:17:42.849234 30565 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:17:42.850613 30565 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:17:42.850684 30565 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:17:42.850768 30565 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:17:42.850811 30565 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:17:42.884500 30565 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.1:40591
I20260925 15:17:42.884536 30692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.1:40591 every 8 connection(s)
I20260925 15:17:42.885875 30565 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349162370577-12320-0/minicluster-data/ts-0/data/info.pb
I20260925 15:17:42.889879 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 30565
I20260925 15:17:42.890076 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349162370577-12320-0/minicluster-data/ts-0/wal/instance
I20260925 15:17:42.890601 12320 external_mini_cluster.cc:1428] Setting key 566fb69bfd31b6bfc8946d79f0b31529
Sep 25 15:17:42 dist-test-slave-p468 krb5kdc[30459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349462, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.1@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
I20260925 15:17:42.907872 30695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:42.888485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.1:45135 (local address 127.12.8.62:46883)
0925 15:17:42.888952 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:42.888959 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:42.889976 (+ 1017us) server_negotiation.cc:407] Connection header received
0925 15:17:42.892020 (+ 2044us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:42.892025 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:42.892121 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:42.892243 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:17:42.893735 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:42.894539 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:42.896082 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:42.896479 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:42.896626 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:42.900903 (+ 4277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:17:42.900908 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:17:42.900911 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:17:42.900945 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:17:42.904410 (+ 3465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:17:42.905165 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:17:42.905173 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:17:42.905179 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:17:42.905330 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:17:42.905781 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:17:42.905788 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:17:42.905794 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:17:42.906169 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:17:42.906380 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:42.907119 (+ 739us) server_negotiation.cc:299] Negotiation successful
0925 15:17:42.907390 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":219,"threads_started":1}
I20260925 15:17:42.908568 30696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:42.889049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:46883 (local address 127.12.8.1:45135)
0925 15:17:42.889780 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:42.889821 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 15:17:42.891738 (+ 1917us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:17:42.892491 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:17:42.892500 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:17:42.893011 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:17:42.893559 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:42.893576 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:42.894699 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:42.894704 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:42.895926 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:42.895938 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:42.896711 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:42.896721 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:42.896961 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:42.897740 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:17:42.897764 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:17:42.900635 (+ 2871us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:17:42.904565 (+ 3930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:17:42.904572 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:17:42.904585 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:17:42.905034 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:17:42.905478 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:17:42.905481 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:17:42.905483 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:17:42.905663 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:17:42.906368 (+ 705us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:17:42.906374 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:17:42.906710 (+ 336us) client_negotiation.cc:769] Sending connection context
0925 15:17:42.906964 (+ 254us) client_negotiation.cc:240] Negotiation successful
0925 15:17:42.907356 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":318,"threads_started":1}
I20260925 15:17:42.910173 30693 heartbeater.cc:347] Connected to a master server at 127.12.8.62:46883
I20260925 15:17:42.910547 30693 heartbeater.cc:464] Registering TS with master...
I20260925 15:17:42.911381 30693 heartbeater.cc:511] Master 127.12.8.62:46883 requested a full tablet report, sending...
I20260925 15:17:42.913515 30504 ts_manager.cc:194] Registered new tserver with Master: 196922ce8c204035b9a27a502b2ffba7 (127.12.8.1:40591)
I20260925 15:17:42.915046 30504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.1@KRBTEST.COM'} at 127.12.8.1:45135
WARNING: no policy specified for kudu/127.12.8.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.2.keytab.
Entry for principal kudu/127.12.8.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.2.keytab.
I20260925 15:17:42.949890 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349162370577-12320-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349162370577-12320-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349162370577-12320-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349162370577-12320-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.2:0
--local_ip_for_outbound_sockets=127.12.8.2
--webserver_interface=127.12.8.2
--webserver_port=0
--tserver_master_addrs=127.12.8.62:46883
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.2.keytab
--principal=kudu/127.12.8.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:38919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:17:43.269708 30699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:17:43.269991 30699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:17:43.270084 30699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:17:43.276950 30699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:17:43.277047 30699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:17:43.277153 30699 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.2
I20260925 15:17:43.286329 30699 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:38919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349162370577-12320-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349162370577-12320-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.2.keytab
--principal=kudu/127.12.8.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349162370577-12320-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.8.2
--webserver_port=0
--tserver_master_addrs=127.12.8.62:46883
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.2
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349162370577-12320-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:17:43.287477 30699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:17:43.288803 30699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:17:43.301669 30704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:17:43.301672 30705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:17:43.302671 30707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:17:43.303006 30699 server_base.cc:1034] running on GCE node
I20260925 15:17:43.303635 30699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:17:43.304590 30699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:17:43.305830 30699 hybrid_clock.cc:648] HybridClock initialized: now 1790349463305784 us; error 38 us; skew 500 ppm
I20260925 15:17:43.306142 30699 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:17:43 dist-test-slave-p468 krb5kdc[30459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349463, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:17:43.312180 30699 init.cc:377] Logged in from keytab as kudu/127.12.8.2@KRBTEST.COM (short username kudu)
I20260925 15:17:43.313555 30699 webserver.cc:460] Webserver started at http://127.12.8.2:42251/ using document root <none> and password file <none>
I20260925 15:17:43.314163 30699 fs_manager.cc:359] Metadata directory not provided
I20260925 15:17:43.314335 30699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:17:43.314594 30699 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:17:43.317245 30699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349162370577-12320-0/minicluster-data/ts-1/data/instance:
uuid: "ffd32ada67cd47368ab806773dadbd44"
format_stamp: "Formatted at 2026-09-25 15:17:43 on dist-test-slave-p468"
server_key: "e4a9294eaedc9fa307716de101b12e42"
server_key_iv: "e52a77b34b7b5b1395ad232a05b4e1a7"
server_key_version: "encryptionkey@0"
I20260925 15:17:43.317924 30699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349162370577-12320-0/minicluster-data/ts-1/wal/instance:
uuid: "ffd32ada67cd47368ab806773dadbd44"
format_stamp: "Formatted at 2026-09-25 15:17:43 on dist-test-slave-p468"
server_key: "e4a9294eaedc9fa307716de101b12e42"
server_key_iv: "e52a77b34b7b5b1395ad232a05b4e1a7"
server_key_version: "encryptionkey@0"
I20260925 15:17:43.322729 30699 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 15:17:43.325855 30714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:17:43.327149 30699 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:17:43.327322 30699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349162370577-12320-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349162370577-12320-0/minicluster-data/ts-1/wal
uuid: "ffd32ada67cd47368ab806773dadbd44"
format_stamp: "Formatted at 2026-09-25 15:17:43 on dist-test-slave-p468"
server_key: "e4a9294eaedc9fa307716de101b12e42"
server_key_iv: "e52a77b34b7b5b1395ad232a05b4e1a7"
server_key_version: "encryptionkey@0"
I20260925 15:17:43.327522 30699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349162370577-12320-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349162370577-12320-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349162370577-12320-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:17:43.357522 30699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:17:43.361654 30699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:17:43.361950 30699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:17:43.363003 30699 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:17:43.364297 30699 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:17:43.364408 30699 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:17:43.364560 30699 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:17:43.364643 30699 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:17:43.396852 30699 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.2:43573
I20260925 15:17:43.396950 30826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.2:43573 every 8 connection(s)
I20260925 15:17:43.398144 30699 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349162370577-12320-0/minicluster-data/ts-1/data/info.pb
I20260925 15:17:43.403470 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 30699
I20260925 15:17:43.403652 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349162370577-12320-0/minicluster-data/ts-1/wal/instance
I20260925 15:17:43.404099 12320 external_mini_cluster.cc:1428] Setting key ce83036484f6b5892d5b47cb2b9b0468
Sep 25 15:17:43 dist-test-slave-p468 krb5kdc[30459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349463, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.2@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
I20260925 15:17:43.419076 30695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:43.400894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.2:59131 (local address 127.12.8.62:46883)
0925 15:17:43.401113 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:43.401119 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:43.402390 (+ 1271us) server_negotiation.cc:407] Connection header received
0925 15:17:43.404290 (+ 1900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:43.404296 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:43.404393 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:43.404519 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:17:43.406213 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:43.406907 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:43.408423 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:43.408853 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:43.409031 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:43.412343 (+ 3312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:17:43.412348 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:17:43.412351 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:17:43.412381 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:17:43.415838 (+ 3457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:17:43.416576 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:17:43.416584 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:17:43.416590 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:17:43.416748 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:17:43.417164 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:17:43.417171 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:17:43.417177 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:17:43.417542 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:17:43.417689 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:43.418380 (+ 691us) server_negotiation.cc:299] Negotiation successful
0925 15:17:43.418637 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260925 15:17:43.419711 30829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:43.401413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:46883 (local address 127.12.8.2:59131)
0925 15:17:43.402170 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:43.402274 (+ 104us) client_negotiation.cc:174] Beginning negotiation
0925 15:17:43.404002 (+ 1728us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:17:43.404743 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:17:43.404751 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:17:43.405376 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:17:43.406037 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:43.406053 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:43.407075 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:43.407080 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:43.408276 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:43.408287 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:43.409074 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:43.409081 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:43.409317 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:43.409997 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:17:43.410018 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:17:43.412127 (+ 2109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:17:43.415988 (+ 3861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:17:43.415994 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:17:43.416005 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:17:43.416445 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:17:43.416861 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:17:43.416864 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:17:43.416866 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:17:43.417046 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:17:43.417661 (+ 615us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:17:43.417666 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:17:43.417971 (+ 305us) client_negotiation.cc:769] Sending connection context
0925 15:17:43.418240 (+ 269us) client_negotiation.cc:240] Negotiation successful
0925 15:17:43.418626 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":317,"threads_started":1}
I20260925 15:17:43.421330 30827 heartbeater.cc:347] Connected to a master server at 127.12.8.62:46883
I20260925 15:17:43.421655 30827 heartbeater.cc:464] Registering TS with master...
I20260925 15:17:43.422533 30827 heartbeater.cc:511] Master 127.12.8.62:46883 requested a full tablet report, sending...
I20260925 15:17:43.424204 30504 ts_manager.cc:194] Registered new tserver with Master: ffd32ada67cd47368ab806773dadbd44 (127.12.8.2:43573)
I20260925 15:17:43.425397 30504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.2@KRBTEST.COM'} at 127.12.8.2:59131
WARNING: no policy specified for kudu/127.12.8.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.3.keytab.
Entry for principal kudu/127.12.8.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.3.keytab.
I20260925 15:17:43.461294 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349162370577-12320-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349162370577-12320-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349162370577-12320-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349162370577-12320-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.3:0
--local_ip_for_outbound_sockets=127.12.8.3
--webserver_interface=127.12.8.3
--webserver_port=0
--tserver_master_addrs=127.12.8.62:46883
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.3.keytab
--principal=kudu/127.12.8.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:38919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:17:43.789475 30832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:17:43.789750 30832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:17:43.789919 30832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:17:43.797842 30832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:17:43.797955 30832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:17:43.798139 30832 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.3
I20260925 15:17:43.807971 30832 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:38919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349162370577-12320-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349162370577-12320-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.3.keytab
--principal=kudu/127.12.8.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349162370577-12320-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.8.3
--webserver_port=0
--tserver_master_addrs=127.12.8.62:46883
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.3
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349162370577-12320-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:17:43.809182 30832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:17:43.810493 30832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:17:43.823637 30840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:17:43.823683 30838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:17:43.823796 30837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:17:43.824832 30832 server_base.cc:1034] running on GCE node
I20260925 15:17:43.825538 30832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:17:43.826536 30832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:17:43.827729 30832 hybrid_clock.cc:648] HybridClock initialized: now 1790349463827667 us; error 56 us; skew 500 ppm
I20260925 15:17:43.828050 30832 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:17:43 dist-test-slave-p468 krb5kdc[30459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349463, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:17:43.831311 30832 init.cc:377] Logged in from keytab as kudu/127.12.8.3@KRBTEST.COM (short username kudu)
I20260925 15:17:43.832876 30832 webserver.cc:460] Webserver started at http://127.12.8.3:37687/ using document root <none> and password file <none>
I20260925 15:17:43.833549 30832 fs_manager.cc:359] Metadata directory not provided
I20260925 15:17:43.833678 30832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:17:43.833988 30832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:17:43.837076 30832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349162370577-12320-0/minicluster-data/ts-2/data/instance:
uuid: "444292353eff41dc8a82548160c067b1"
format_stamp: "Formatted at 2026-09-25 15:17:43 on dist-test-slave-p468"
server_key: "81170cc397ff334b0fd37b555bc980ac"
server_key_iv: "1c50a967627db91e53bb6fd71d32c0ff"
server_key_version: "encryptionkey@0"
I20260925 15:17:43.837803 30832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349162370577-12320-0/minicluster-data/ts-2/wal/instance:
uuid: "444292353eff41dc8a82548160c067b1"
format_stamp: "Formatted at 2026-09-25 15:17:43 on dist-test-slave-p468"
server_key: "81170cc397ff334b0fd37b555bc980ac"
server_key_iv: "1c50a967627db91e53bb6fd71d32c0ff"
server_key_version: "encryptionkey@0"
I20260925 15:17:43.842978 30832 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260925 15:17:43.846329 30847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:17:43.847712 30832 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 15:17:43.847971 30832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349162370577-12320-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349162370577-12320-0/minicluster-data/ts-2/wal
uuid: "444292353eff41dc8a82548160c067b1"
format_stamp: "Formatted at 2026-09-25 15:17:43 on dist-test-slave-p468"
server_key: "81170cc397ff334b0fd37b555bc980ac"
server_key_iv: "1c50a967627db91e53bb6fd71d32c0ff"
server_key_version: "encryptionkey@0"
I20260925 15:17:43.848186 30832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349162370577-12320-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349162370577-12320-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349162370577-12320-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:17:43.869779 30832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:17:43.874073 30832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:17:43.874409 30832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:17:43.875341 30832 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:17:43.876704 30832 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:17:43.876776 30832 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:17:43.876950 30832 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:17:43.877003 30832 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:17:43.910815 30832 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.3:45215
I20260925 15:17:43.910859 30959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.3:45215 every 8 connection(s)
I20260925 15:17:43.912086 30832 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349162370577-12320-0/minicluster-data/ts-2/data/info.pb
I20260925 15:17:43.914345 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 30832
I20260925 15:17:43.914548 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349162370577-12320-0/minicluster-data/ts-2/wal/instance
I20260925 15:17:43.915091 12320 external_mini_cluster.cc:1428] Setting key ab3d26e9bdd5196125f9517f71e3aa86
I20260925 15:17:43.918385 30693 heartbeater.cc:503] Master 127.12.8.62:46883 was elected leader, sending a full tablet report...
Sep 25 15:17:43 dist-test-slave-p468 krb5kdc[30459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349463, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.3@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
I20260925 15:17:43.933156 30695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:43.914671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.3:39335 (local address 127.12.8.62:46883)
0925 15:17:43.914851 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:43.914856 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:43.916309 (+ 1453us) server_negotiation.cc:407] Connection header received
0925 15:17:43.918788 (+ 2479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:43.918792 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:43.919174 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:43.919307 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:17:43.921054 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:43.921847 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:43.923380 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:43.923759 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:43.923861 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:43.927035 (+ 3174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:17:43.927039 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:17:43.927042 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:17:43.927073 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:17:43.929972 (+ 2899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:17:43.930959 (+ 987us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:17:43.930964 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:17:43.930966 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:17:43.931105 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:17:43.931486 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:17:43.931489 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:17:43.931491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:17:43.931760 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:17:43.931936 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:43.932664 (+ 728us) server_negotiation.cc:299] Negotiation successful
0925 15:17:43.932821 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 15:17:43.934020 30962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:43.915327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:46883 (local address 127.12.8.3:39335)
0925 15:17:43.916161 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:43.916208 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 15:17:43.918504 (+ 2296us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:17:43.919610 (+ 1106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:17:43.919620 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:17:43.920333 (+ 713us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:17:43.920880 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:43.920897 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:43.921978 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:43.921982 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:43.923235 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:43.923246 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:43.923947 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:43.923951 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:43.924141 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:43.924754 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:17:43.924779 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:17:43.926853 (+ 2074us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:17:43.930132 (+ 3279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:17:43.930141 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:17:43.930157 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:17:43.930843 (+ 686us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:17:43.931200 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:17:43.931203 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:17:43.931205 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:17:43.931368 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:17:43.931909 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:17:43.931915 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:17:43.932313 (+ 398us) client_negotiation.cc:769] Sending connection context
0925 15:17:43.932551 (+ 238us) client_negotiation.cc:240] Negotiation successful
0925 15:17:43.932899 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":335,"threads_started":1}
I20260925 15:17:43.935359 30960 heartbeater.cc:347] Connected to a master server at 127.12.8.62:46883
I20260925 15:17:43.935643 30960 heartbeater.cc:464] Registering TS with master...
I20260925 15:17:43.936411 30960 heartbeater.cc:511] Master 127.12.8.62:46883 requested a full tablet report, sending...
I20260925 15:17:43.937701 30504 ts_manager.cc:194] Registered new tserver with Master: 444292353eff41dc8a82548160c067b1 (127.12.8.3:45215)
I20260925 15:17:43.938526 30504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.3@KRBTEST.COM'} at 127.12.8.3:39335
I20260925 15:17:43.942952 12320 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:17:43.960048 30695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:43.946956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53422 (local address 127.12.8.62:46883)
0925 15:17:43.947145 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:43.947153 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:43.947602 (+ 449us) server_negotiation.cc:407] Connection header received
0925 15:17:43.947806 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:43.947816 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:43.947880 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:43.947988 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:17:43.949382 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:43.950096 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:43.951545 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:43.951945 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:43.952103 (+ 158us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:43.954743 (+ 2640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:17:43.954749 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:17:43.954760 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:17:43.954789 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:17:43.957112 (+ 2323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:17:43.957866 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:17:43.957871 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:17:43.957875 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:17:43.958003 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:17:43.958561 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:17:43.958567 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:17:43.958570 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:17:43.958865 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:17:43.959164 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:43.959585 (+ 421us) server_negotiation.cc:299] Negotiation successful
0925 15:17:43.959765 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 15:17:43.964061 30504 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53422:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:17:43.970832 30504 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:17:44.004927 30974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:43.983951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.2:43573 (local address 127.0.0.1:37458)
0925 15:17:43.996918 (+ 12967us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:43.996938 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 15:17:43.997118 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:17:43.998071 (+ 953us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:17:43.998079 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:17:43.998113 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:17:43.998627 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:43.998643 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:44.000227 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:44.000237 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:44.001981 (+ 1744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:44.001993 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:44.003709 (+ 1716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:44.003715 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:44.004070 (+ 355us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:44.004137 (+ 67us) client_negotiation.cc:769] Sending connection context
0925 15:17:44.004274 (+ 137us) client_negotiation.cc:240] Negotiation successful
0925 15:17:44.004495 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12777,"mutex_wait_us":85,"thread_start_us":11923,"threads_started":1}
I20260925 15:17:44.004925 30975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:43.983232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55514 (local address 127.12.8.3:45215)
0925 15:17:43.995553 (+ 12321us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:43.995565 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:43.996396 (+ 831us) server_negotiation.cc:407] Connection header received
0925 15:17:43.996628 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:43.996637 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:43.996914 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:43.997149 (+ 235us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:17:43.997816 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:43.999101 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:44.002095 (+ 2994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:44.003574 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:44.004117 (+ 543us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:44.004327 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:44.004397 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 15:17:44.004561 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12086,"thread_start_us":12478,"threads_started":1}
I20260925 15:17:44.005713 30976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:43.982899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.3:45215 (local address 127.0.0.1:55514)
0925 15:17:43.996223 (+ 13324us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:43.996250 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 15:17:43.996433 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:17:43.997165 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:17:43.997173 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:17:43.997205 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:17:43.997616 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:43.997628 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:43.999279 (+ 1651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:43.999285 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:44.001916 (+ 2631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:44.001931 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:44.004020 (+ 2089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:44.004026 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:44.004228 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:44.004266 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 15:17:44.004330 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 15:17:44.005333 (+ 1003us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13146,"mutex_wait_us":866,"thread_start_us":13652,"threads_started":1}
I20260925 15:17:44.005879 30977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:43.984363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37458 (local address 127.12.8.2:43573)
0925 15:17:43.995433 (+ 11070us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:43.995446 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:43.997026 (+ 1580us) server_negotiation.cc:407] Connection header received
0925 15:17:43.997300 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:43.997307 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:43.997530 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:43.997722 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:17:43.998828 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:44.000066 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:44.002211 (+ 2145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:44.003490 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:44.003654 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:44.003860 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:44.005370 (+ 1510us) server_negotiation.cc:299] Negotiation successful
0925 15:17:44.005555 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10828,"thread_start_us":11058,"threads_started":1}
I20260925 15:17:44.007315 30972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:43.982497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.1:40591 (local address 127.0.0.1:34036)
0925 15:17:43.983489 (+ 992us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:43.983543 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 15:17:43.983746 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:17:43.999879 (+ 16133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:17:43.999884 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:17:43.999909 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:17:44.000309 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:44.000321 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:44.002426 (+ 2105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:44.002430 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:44.004521 (+ 2091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:44.004535 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:44.006474 (+ 1939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:44.006481 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:44.006661 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:44.006692 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 15:17:44.006803 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 15:17:44.006937 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":687,"mutex_wait_us":47,"thread_start_us":609,"threads_started":1}
I20260925 15:17:44.008108 30973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:43.982784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34036 (local address 127.12.8.1:40591)
0925 15:17:43.999313 (+ 16529us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:43.999324 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:43.999350 (+ 26us) server_negotiation.cc:407] Connection header received
0925 15:17:43.999447 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:43.999456 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:43.999655 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:44.000543 (+ 888us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:17:44.000793 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:44.002294 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:44.004710 (+ 2416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:44.006278 (+ 1568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:44.006400 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:44.006623 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:44.007632 (+ 1009us) server_negotiation.cc:299] Negotiation successful
0925 15:17:44.007810 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16362,"thread_start_us":16223,"threads_started":1}
I20260925 15:17:44.009550 30762 tablet_service.cc:1467] Processing CreateTablet for tablet 9b0f9fd2e79b4e59949f5ef3439be894 (DEFAULT_TABLE table=test-table [id=6efe80bbb4f74f009fc6c0afe7cdf7a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:17:44.009555 30895 tablet_service.cc:1467] Processing CreateTablet for tablet 9b0f9fd2e79b4e59949f5ef3439be894 (DEFAULT_TABLE table=test-table [id=6efe80bbb4f74f009fc6c0afe7cdf7a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:17:44.010946 30895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b0f9fd2e79b4e59949f5ef3439be894. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:17:44.011022 30762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b0f9fd2e79b4e59949f5ef3439be894. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:17:44.011968 30628 tablet_service.cc:1467] Processing CreateTablet for tablet 9b0f9fd2e79b4e59949f5ef3439be894 (DEFAULT_TABLE table=test-table [id=6efe80bbb4f74f009fc6c0afe7cdf7a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:17:44.013334 30628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b0f9fd2e79b4e59949f5ef3439be894. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:17:44.023624 30978 tablet_bootstrap.cc:492] T 9b0f9fd2e79b4e59949f5ef3439be894 P ffd32ada67cd47368ab806773dadbd44: Bootstrap starting.
I20260925 15:17:44.025651 30979 tablet_bootstrap.cc:492] T 9b0f9fd2e79b4e59949f5ef3439be894 P 444292353eff41dc8a82548160c067b1: Bootstrap starting.
I20260925 15:17:44.027374 30978 tablet_bootstrap.cc:654] T 9b0f9fd2e79b4e59949f5ef3439be894 P ffd32ada67cd47368ab806773dadbd44: Neither blocks nor log segments found. Creating new log.
I20260925 15:17:44.028918 30978 log.cc:824] T 9b0f9fd2e79b4e59949f5ef3439be894 P ffd32ada67cd47368ab806773dadbd44: Log is configured to *not* fsync() on all Append() calls
I20260925 15:17:44.029739 30979 tablet_bootstrap.cc:654] T 9b0f9fd2e79b4e59949f5ef3439be894 P 444292353eff41dc8a82548160c067b1: Neither blocks nor log segments found. Creating new log.
I20260925 15:17:44.030346 30980 tablet_bootstrap.cc:492] T 9b0f9fd2e79b4e59949f5ef3439be894 P 196922ce8c204035b9a27a502b2ffba7: Bootstrap starting.
I20260925 15:17:44.031031 30979 log.cc:824] T 9b0f9fd2e79b4e59949f5ef3439be894 P 444292353eff41dc8a82548160c067b1: Log is configured to *not* fsync() on all Append() calls
I20260925 15:17:44.032284 30978 tablet_bootstrap.cc:492] T 9b0f9fd2e79b4e59949f5ef3439be894 P ffd32ada67cd47368ab806773dadbd44: No bootstrap required, opened a new log
I20260925 15:17:44.032578 30978 ts_tablet_manager.cc:1397] T 9b0f9fd2e79b4e59949f5ef3439be894 P ffd32ada67cd47368ab806773dadbd44: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260925 15:17:44.033991 30980 tablet_bootstrap.cc:654] T 9b0f9fd2e79b4e59949f5ef3439be894 P 196922ce8c204035b9a27a502b2ffba7: Neither blocks nor log segments found. Creating new log.
I20260925 15:17:44.034225 30979 tablet_bootstrap.cc:492] T 9b0f9fd2e79b4e59949f5ef3439be894 P 444292353eff41dc8a82548160c067b1: No bootstrap required, opened a new log
I20260925 15:17:44.034469 30979 ts_tablet_manager.cc:1397] T 9b0f9fd2e79b4e59949f5ef3439be894 P 444292353eff41dc8a82548160c067b1: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.002s
I20260925 15:17:44.035091 30980 log.cc:824] T 9b0f9fd2e79b4e59949f5ef3439be894 P 196922ce8c204035b9a27a502b2ffba7: Log is configured to *not* fsync() on all Append() calls
I20260925 15:17:44.037899 30980 tablet_bootstrap.cc:492] T 9b0f9fd2e79b4e59949f5ef3439be894 P 196922ce8c204035b9a27a502b2ffba7: No bootstrap required, opened a new log
I20260925 15:17:44.038158 30980 ts_tablet_manager.cc:1397] T 9b0f9fd2e79b4e59949f5ef3439be894 P 196922ce8c204035b9a27a502b2ffba7: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260925 15:17:44.043460 30978 raft_consensus.cc:348] T 9b0f9fd2e79b4e59949f5ef3439be894 P ffd32ada67cd47368ab806773dadbd44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "196922ce8c204035b9a27a502b2ffba7" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 40591 } } peers { permanent_uuid: "444292353eff41dc8a82548160c067b1" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 45215 } } peers { permanent_uuid: "ffd32ada67cd47368ab806773dadbd44" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 43573 } }
I20260925 15:17:44.043895 30978 raft_consensus.cc:374] T 9b0f9fd2e79b4e59949f5ef3439be894 P ffd32ada67cd47368ab806773dadbd44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:17:44.044011 30978 raft_consensus.cc:729] T 9b0f9fd2e79b4e59949f5ef3439be894 P ffd32ada67cd47368ab806773dadbd44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffd32ada67cd47368ab806773dadbd44, State: Initialized, Role: FOLLOWER
I20260925 15:17:44.044659 30978 consensus_queue.cc:260] T 9b0f9fd2e79b4e59949f5ef3439be894 P ffd32ada67cd47368ab806773dadbd44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "196922ce8c204035b9a27a502b2ffba7" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 40591 } } peers { permanent_uuid: "444292353eff41dc8a82548160c067b1" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 45215 } } peers { permanent_uuid: "ffd32ada67cd47368ab806773dadbd44" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 43573 } }
I20260925 15:17:44.044893 30979 raft_consensus.cc:348] T 9b0f9fd2e79b4e59949f5ef3439be894 P 444292353eff41dc8a82548160c067b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "196922ce8c204035b9a27a502b2ffba7" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 40591 } } peers { permanent_uuid: "444292353eff41dc8a82548160c067b1" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 45215 } } peers { permanent_uuid: "ffd32ada67cd47368ab806773dadbd44" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 43573 } }
I20260925 15:17:44.045404 30979 raft_consensus.cc:374] T 9b0f9fd2e79b4e59949f5ef3439be894 P 444292353eff41dc8a82548160c067b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:17:44.045511 30979 raft_consensus.cc:729] T 9b0f9fd2e79b4e59949f5ef3439be894 P 444292353eff41dc8a82548160c067b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 444292353eff41dc8a82548160c067b1, State: Initialized, Role: FOLLOWER
I20260925 15:17:44.045961 30827 heartbeater.cc:503] Master 127.12.8.62:46883 was elected leader, sending a full tablet report...
I20260925 15:17:44.045821 30980 raft_consensus.cc:348] T 9b0f9fd2e79b4e59949f5ef3439be894 P 196922ce8c204035b9a27a502b2ffba7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "196922ce8c204035b9a27a502b2ffba7" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 40591 } } peers { permanent_uuid: "444292353eff41dc8a82548160c067b1" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 45215 } } peers { permanent_uuid: "ffd32ada67cd47368ab806773dadbd44" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 43573 } }
I20260925 15:17:44.046238 30980 raft_consensus.cc:374] T 9b0f9fd2e79b4e59949f5ef3439be894 P 196922ce8c204035b9a27a502b2ffba7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:17:44.046147 30979 consensus_queue.cc:260] T 9b0f9fd2e79b4e59949f5ef3439be894 P 444292353eff41dc8a82548160c067b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "196922ce8c204035b9a27a502b2ffba7" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 40591 } } peers { permanent_uuid: "444292353eff41dc8a82548160c067b1" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 45215 } } peers { permanent_uuid: "ffd32ada67cd47368ab806773dadbd44" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 43573 } }
I20260925 15:17:44.046336 30980 raft_consensus.cc:729] T 9b0f9fd2e79b4e59949f5ef3439be894 P 196922ce8c204035b9a27a502b2ffba7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 196922ce8c204035b9a27a502b2ffba7, State: Initialized, Role: FOLLOWER
I20260925 15:17:44.046525 30978 ts_tablet_manager.cc:1428] T 9b0f9fd2e79b4e59949f5ef3439be894 P ffd32ada67cd47368ab806773dadbd44: Time spent starting tablet: real 0.014s user 0.008s sys 0.005s
I20260925 15:17:44.046859 30980 consensus_queue.cc:260] T 9b0f9fd2e79b4e59949f5ef3439be894 P 196922ce8c204035b9a27a502b2ffba7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "196922ce8c204035b9a27a502b2ffba7" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 40591 } } peers { permanent_uuid: "444292353eff41dc8a82548160c067b1" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 45215 } } peers { permanent_uuid: "ffd32ada67cd47368ab806773dadbd44" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 43573 } }
I20260925 15:17:44.048522 30960 heartbeater.cc:503] Master 127.12.8.62:46883 was elected leader, sending a full tablet report...
I20260925 15:17:44.049144 30979 ts_tablet_manager.cc:1428] T 9b0f9fd2e79b4e59949f5ef3439be894 P 444292353eff41dc8a82548160c067b1: Time spent starting tablet: real 0.015s user 0.012s sys 0.002s
I20260925 15:17:44.051515 30980 ts_tablet_manager.cc:1428] T 9b0f9fd2e79b4e59949f5ef3439be894 P 196922ce8c204035b9a27a502b2ffba7: Time spent starting tablet: real 0.013s user 0.006s sys 0.007s
I20260925 15:17:44.093474 30984 raft_consensus.cc:482] T 9b0f9fd2e79b4e59949f5ef3439be894 P ffd32ada67cd47368ab806773dadbd44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:17:44.093751 30984 raft_consensus.cc:504] T 9b0f9fd2e79b4e59949f5ef3439be894 P ffd32ada67cd47368ab806773dadbd44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "196922ce8c204035b9a27a502b2ffba7" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 40591 } } peers { permanent_uuid: "444292353eff41dc8a82548160c067b1" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 45215 } } peers { permanent_uuid: "ffd32ada67cd47368ab806773dadbd44" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 43573 } }
I20260925 15:17:44.095428 30984 leader_election.cc:288] T 9b0f9fd2e79b4e59949f5ef3439be894 P ffd32ada67cd47368ab806773dadbd44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 196922ce8c204035b9a27a502b2ffba7 (127.12.8.1:40591), 444292353eff41dc8a82548160c067b1 (127.12.8.3:45215)
I20260925 15:17:44.102799 30973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:44.095993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.2:33197 (local address 127.12.8.1:40591)
0925 15:17:44.096310 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:44.096316 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:44.096785 (+ 469us) server_negotiation.cc:407] Connection header received
0925 15:17:44.097017 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:44.097027 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:44.097084 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:44.097203 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:17:44.097957 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:44.098961 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:44.100799 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:44.101739 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:44.101844 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:44.101916 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:44.102409 (+ 493us) server_negotiation.cc:299] Negotiation successful
0925 15:17:44.102559 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142}
I20260925 15:17:44.102998 30975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:44.095830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.2:54173 (local address 127.12.8.3:45215)
0925 15:17:44.096095 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:44.096104 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:44.096879 (+ 775us) server_negotiation.cc:407] Connection header received
0925 15:17:44.097076 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:44.097100 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:44.097157 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:44.097269 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:17:44.098208 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:44.099287 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:44.101054 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:44.101922 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:44.102058 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:44.102158 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:44.102614 (+ 456us) server_negotiation.cc:299] Negotiation successful
0925 15:17:44.102738 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 15:17:44.103094 30987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:44.095673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.1:40591 (local address 127.12.8.2:33197)
0925 15:17:44.096653 (+ 980us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:44.096675 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 15:17:44.096864 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:17:44.097245 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:17:44.097253 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:17:44.097281 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:17:44.097793 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:44.097808 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:44.099093 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:44.099099 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:44.100637 (+ 1538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:44.100649 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:44.101980 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:44.101988 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:44.102182 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:44.102271 (+ 89us) client_negotiation.cc:769] Sending connection context
0925 15:17:44.102356 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 15:17:44.102448 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":806,"mutex_wait_us":26,"thread_start_us":564,"threads_started":1}
I20260925 15:17:44.103067 30988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:44.095673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.3:45215 (local address 127.12.8.2:54173)
0925 15:17:44.096768 (+ 1095us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:44.096781 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:17:44.096934 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:17:44.097390 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:17:44.097395 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:17:44.097431 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:17:44.097952 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:44.097965 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:44.099422 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:44.099426 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:44.100867 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:44.100880 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:44.102170 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:44.102174 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:44.102438 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:44.102468 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 15:17:44.102557 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 15:17:44.102672 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":983,"mutex_wait_us":137,"thread_start_us":709,"threads_started":1}
I20260925 15:17:44.103506 30985 raft_consensus.cc:482] T 9b0f9fd2e79b4e59949f5ef3439be894 P 444292353eff41dc8a82548160c067b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:17:44.103760 30985 raft_consensus.cc:504] T 9b0f9fd2e79b4e59949f5ef3439be894 P 444292353eff41dc8a82548160c067b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "196922ce8c204035b9a27a502b2ffba7" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 40591 } } peers { permanent_uuid: "444292353eff41dc8a82548160c067b1" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 45215 } } peers { permanent_uuid: "ffd32ada67cd47368ab806773dadbd44" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 43573 } }
I20260925 15:17:44.104235 30915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0f9fd2e79b4e59949f5ef3439be894" candidate_uuid: "ffd32ada67cd47368ab806773dadbd44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "444292353eff41dc8a82548160c067b1" is_pre_election: true
I20260925 15:17:44.104720 30915 raft_consensus.cc:2454] T 9b0f9fd2e79b4e59949f5ef3439be894 P 444292353eff41dc8a82548160c067b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ffd32ada67cd47368ab806773dadbd44 in term 0.
I20260925 15:17:44.104722 30648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0f9fd2e79b4e59949f5ef3439be894" candidate_uuid: "ffd32ada67cd47368ab806773dadbd44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "196922ce8c204035b9a27a502b2ffba7" is_pre_election: true
I20260925 15:17:44.105260 30648 raft_consensus.cc:2454] T 9b0f9fd2e79b4e59949f5ef3439be894 P 196922ce8c204035b9a27a502b2ffba7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ffd32ada67cd47368ab806773dadbd44 in term 0.
I20260925 15:17:44.105460 30715 leader_election.cc:302] T 9b0f9fd2e79b4e59949f5ef3439be894 P ffd32ada67cd47368ab806773dadbd44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 444292353eff41dc8a82548160c067b1, ffd32ada67cd47368ab806773dadbd44; no voters:
I20260925 15:17:44.105656 30985 leader_election.cc:288] T 9b0f9fd2e79b4e59949f5ef3439be894 P 444292353eff41dc8a82548160c067b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 196922ce8c204035b9a27a502b2ffba7 (127.12.8.1:40591), ffd32ada67cd47368ab806773dadbd44 (127.12.8.2:43573)
I20260925 15:17:44.105862 30984 raft_consensus.cc:2781] T 9b0f9fd2e79b4e59949f5ef3439be894 P ffd32ada67cd47368ab806773dadbd44 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:17:44.106026 30984 raft_consensus.cc:482] T 9b0f9fd2e79b4e59949f5ef3439be894 P ffd32ada67cd47368ab806773dadbd44 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:17:44.106133 30984 raft_consensus.cc:3037] T 9b0f9fd2e79b4e59949f5ef3439be894 P ffd32ada67cd47368ab806773dadbd44 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:17:44.109506 30984 raft_consensus.cc:504] T 9b0f9fd2e79b4e59949f5ef3439be894 P ffd32ada67cd47368ab806773dadbd44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "196922ce8c204035b9a27a502b2ffba7" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 40591 } } peers { permanent_uuid: "444292353eff41dc8a82548160c067b1" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 45215 } } peers { permanent_uuid: "ffd32ada67cd47368ab806773dadbd44" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 43573 } }
I20260925 15:17:44.110520 30984 leader_election.cc:288] T 9b0f9fd2e79b4e59949f5ef3439be894 P ffd32ada67cd47368ab806773dadbd44 [CANDIDATE]: Term 1 election: Requested vote from peers 196922ce8c204035b9a27a502b2ffba7 (127.12.8.1:40591), 444292353eff41dc8a82548160c067b1 (127.12.8.3:45215)
I20260925 15:17:44.110545 30648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0f9fd2e79b4e59949f5ef3439be894" candidate_uuid: "ffd32ada67cd47368ab806773dadbd44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "196922ce8c204035b9a27a502b2ffba7"
I20260925 15:17:44.110641 30915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0f9fd2e79b4e59949f5ef3439be894" candidate_uuid: "ffd32ada67cd47368ab806773dadbd44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "444292353eff41dc8a82548160c067b1"
I20260925 15:17:44.110738 30648 raft_consensus.cc:3037] T 9b0f9fd2e79b4e59949f5ef3439be894 P 196922ce8c204035b9a27a502b2ffba7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:17:44.110829 30915 raft_consensus.cc:3037] T 9b0f9fd2e79b4e59949f5ef3439be894 P 444292353eff41dc8a82548160c067b1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:17:44.113170 30977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:44.106795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.3:41009 (local address 127.12.8.2:43573)
0925 15:17:44.106986 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:44.106991 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:44.107008 (+ 17us) server_negotiation.cc:407] Connection header received
0925 15:17:44.107068 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:44.107072 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:44.107133 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:44.107219 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:17:44.107952 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:44.109015 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:44.111388 (+ 2373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:44.112181 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:44.112342 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:44.112432 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:44.112793 (+ 361us) server_negotiation.cc:299] Negotiation successful
0925 15:17:44.112913 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 15:17:44.113260 30989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:44.106113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.2:43573 (local address 127.12.8.3:41009)
0925 15:17:44.106726 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:44.106742 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:17:44.106843 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:17:44.107377 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:17:44.107382 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:17:44.107405 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:17:44.107780 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:44.107792 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:44.109211 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:44.109218 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:44.111239 (+ 2021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:44.111251 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:44.112390 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:44.112396 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:44.112608 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:44.112646 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 15:17:44.112771 (+ 125us) client_negotiation.cc:240] Negotiation successful
0925 15:17:44.112910 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":326,"threads_started":1}
I20260925 15:17:44.113937 30648 raft_consensus.cc:2454] T 9b0f9fd2e79b4e59949f5ef3439be894 P 196922ce8c204035b9a27a502b2ffba7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ffd32ada67cd47368ab806773dadbd44 in term 1.
I20260925 15:17:44.114022 30915 raft_consensus.cc:2454] T 9b0f9fd2e79b4e59949f5ef3439be894 P 444292353eff41dc8a82548160c067b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ffd32ada67cd47368ab806773dadbd44 in term 1.
I20260925 15:17:44.114373 30718 leader_election.cc:302] T 9b0f9fd2e79b4e59949f5ef3439be894 P ffd32ada67cd47368ab806773dadbd44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 196922ce8c204035b9a27a502b2ffba7, ffd32ada67cd47368ab806773dadbd44; no voters:
I20260925 15:17:44.114673 30984 raft_consensus.cc:2781] T 9b0f9fd2e79b4e59949f5ef3439be894 P ffd32ada67cd47368ab806773dadbd44 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:17:44.115252 30984 raft_consensus.cc:686] T 9b0f9fd2e79b4e59949f5ef3439be894 P ffd32ada67cd47368ab806773dadbd44 [term 1 LEADER]: Becoming Leader. State: Replica: ffd32ada67cd47368ab806773dadbd44, State: Running, Role: LEADER
I20260925 15:17:44.115330 30782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0f9fd2e79b4e59949f5ef3439be894" candidate_uuid: "444292353eff41dc8a82548160c067b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffd32ada67cd47368ab806773dadbd44" is_pre_election: true
I20260925 15:17:44.115748 30962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:44.105781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.1:40591 (local address 127.12.8.3:57017)
0925 15:17:44.105932 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:44.105950 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 15:17:44.106058 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:17:44.107210 (+ 1152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:17:44.107214 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:17:44.107238 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:17:44.107399 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:44.107406 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:44.108377 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:44.108382 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:44.109584 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:44.109598 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:44.114973 (+ 5375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:44.114982 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:44.115240 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:44.115282 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 15:17:44.115374 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 15:17:44.115507 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260925 15:17:44.115805 30984 consensus_queue.cc:237] T 9b0f9fd2e79b4e59949f5ef3439be894 P ffd32ada67cd47368ab806773dadbd44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "196922ce8c204035b9a27a502b2ffba7" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 40591 } } peers { permanent_uuid: "444292353eff41dc8a82548160c067b1" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 45215 } } peers { permanent_uuid: "ffd32ada67cd47368ab806773dadbd44" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 43573 } }
I20260925 15:17:44.116289 30973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:44.106535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.3:57017 (local address 127.12.8.1:40591)
0925 15:17:44.106969 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:44.106974 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:44.106989 (+ 15us) server_negotiation.cc:407] Connection header received
0925 15:17:44.107043 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:44.107046 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:44.107092 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:44.107193 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:17:44.107542 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:44.108238 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:44.109811 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:44.114804 (+ 4993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:44.114934 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:44.115040 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:44.115924 (+ 884us) server_negotiation.cc:299] Negotiation successful
0925 15:17:44.116061 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329}
I20260925 15:17:44.117230 30648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0f9fd2e79b4e59949f5ef3439be894" candidate_uuid: "444292353eff41dc8a82548160c067b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "196922ce8c204035b9a27a502b2ffba7" is_pre_election: true
I20260925 15:17:44.117440 30648 raft_consensus.cc:2379] T 9b0f9fd2e79b4e59949f5ef3439be894 P 196922ce8c204035b9a27a502b2ffba7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 444292353eff41dc8a82548160c067b1 in current term 1: Already voted for candidate ffd32ada67cd47368ab806773dadbd44 in this term.
I20260925 15:17:44.117969 30851 leader_election.cc:302] T 9b0f9fd2e79b4e59949f5ef3439be894 P 444292353eff41dc8a82548160c067b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 444292353eff41dc8a82548160c067b1; no voters: 196922ce8c204035b9a27a502b2ffba7, ffd32ada67cd47368ab806773dadbd44
I20260925 15:17:44.118466 30985 raft_consensus.cc:2726] T 9b0f9fd2e79b4e59949f5ef3439be894 P 444292353eff41dc8a82548160c067b1 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 15:17:44.120640 30504 catalog_manager.cc:5495] T 9b0f9fd2e79b4e59949f5ef3439be894 P ffd32ada67cd47368ab806773dadbd44 reported cstate change: term changed from 0 to 1, leader changed from <none> to ffd32ada67cd47368ab806773dadbd44 (127.12.8.2). New cstate: current_term: 1 leader_uuid: "ffd32ada67cd47368ab806773dadbd44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "196922ce8c204035b9a27a502b2ffba7" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 40591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "444292353eff41dc8a82548160c067b1" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 45215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ffd32ada67cd47368ab806773dadbd44" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 43573 } health_report { overall_health: HEALTHY } } }
W20260925 15:17:44.138365 30694 tablet.cc:2369] T 9b0f9fd2e79b4e59949f5ef3439be894 P 196922ce8c204035b9a27a502b2ffba7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:17:44.150471 30828 tablet.cc:2369] T 9b0f9fd2e79b4e59949f5ef3439be894 P ffd32ada67cd47368ab806773dadbd44: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:17:44.163692 30961 tablet.cc:2369] T 9b0f9fd2e79b4e59949f5ef3439be894 P 444292353eff41dc8a82548160c067b1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:17:44.164553 30977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:44.158935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37466 (local address 127.12.8.2:43573)
0925 15:17:44.159129 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:44.159137 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:44.159161 (+ 24us) server_negotiation.cc:407] Connection header received
0925 15:17:44.159409 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:44.159414 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:44.159521 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:44.159680 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:17:44.160078 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:44.160796 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:44.162153 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:44.162634 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:44.162772 (+ 138us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:44.163359 (+ 587us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:17:44.163810 (+ 451us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:17:44.164008 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:44.164198 (+ 190us) server_negotiation.cc:299] Negotiation successful
0925 15:17:44.164234 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 15:17:44.172819 30648 raft_consensus.cc:1261] T 9b0f9fd2e79b4e59949f5ef3439be894 P 196922ce8c204035b9a27a502b2ffba7 [term 1 FOLLOWER]: Refusing update from remote peer ffd32ada67cd47368ab806773dadbd44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:17:44.172851 30915 raft_consensus.cc:1261] T 9b0f9fd2e79b4e59949f5ef3439be894 P 444292353eff41dc8a82548160c067b1 [term 1 FOLLOWER]: Refusing update from remote peer ffd32ada67cd47368ab806773dadbd44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:17:44.173851 30984 consensus_queue.cc:1035] T 9b0f9fd2e79b4e59949f5ef3439be894 P ffd32ada67cd47368ab806773dadbd44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "444292353eff41dc8a82548160c067b1" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 45215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:17:44.174178 30990 consensus_queue.cc:1035] T 9b0f9fd2e79b4e59949f5ef3439be894 P ffd32ada67cd47368ab806773dadbd44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "196922ce8c204035b9a27a502b2ffba7" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 40591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:17:44.188009 30993 mvcc.cc:205] Tried to move back new op lower bound from 7333271405241159680 to 7333271405023354880. Current Snapshot: MvccSnapshot[applied={T|T < 7333271405241159680}]
I20260925 15:17:44.199219 30503 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:17:44.199443 30503 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:17:44.202961 30503 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3e75f552d2574b20b5446e426140cd6b: Sending DeleteTablet for 3 replicas of tablet 9b0f9fd2e79b4e59949f5ef3439be894
I20260925 15:17:44.204021 30895 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b0f9fd2e79b4e59949f5ef3439be894 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:17:44 UTC) from {username='kudu', principal='kudu/127.12.8.62@KRBTEST.COM'} at 127.0.0.1:55514
I20260925 15:17:44.204095 30628 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b0f9fd2e79b4e59949f5ef3439be894 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:17:44 UTC) from {username='kudu', principal='kudu/127.12.8.62@KRBTEST.COM'} at 127.0.0.1:34036
I20260925 15:17:44.204883 31002 tablet_replica.cc:330] T 9b0f9fd2e79b4e59949f5ef3439be894 P 444292353eff41dc8a82548160c067b1: stopping tablet replica
I20260925 15:17:44.205330 31002 raft_consensus.cc:2229] T 9b0f9fd2e79b4e59949f5ef3439be894 P 444292353eff41dc8a82548160c067b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:17:44.205380 30762 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b0f9fd2e79b4e59949f5ef3439be894 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:17:44 UTC) from {username='kudu', principal='kudu/127.12.8.62@KRBTEST.COM'} at 127.0.0.1:37458
I20260925 15:17:44.205976 31002 raft_consensus.cc:2258] T 9b0f9fd2e79b4e59949f5ef3439be894 P 444292353eff41dc8a82548160c067b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:17:44.211675 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 30565
I20260925 15:17:44.221674 31003 tablet_replica.cc:330] T 9b0f9fd2e79b4e59949f5ef3439be894 P 196922ce8c204035b9a27a502b2ffba7: stopping tablet replica
I20260925 15:17:44.222249 31003 raft_consensus.cc:2229] T 9b0f9fd2e79b4e59949f5ef3439be894 P 196922ce8c204035b9a27a502b2ffba7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:17:44.222796 31003 raft_consensus.cc:2258] T 9b0f9fd2e79b4e59949f5ef3439be894 P 196922ce8c204035b9a27a502b2ffba7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:17:44.222883 31002 ts_tablet_manager.cc:1905] T 9b0f9fd2e79b4e59949f5ef3439be894 P 444292353eff41dc8a82548160c067b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:17:44.227022 31003 ts_tablet_manager.cc:1905] T 9b0f9fd2e79b4e59949f5ef3439be894 P 196922ce8c204035b9a27a502b2ffba7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:17:44.231861 31004 tablet_replica.cc:330] T 9b0f9fd2e79b4e59949f5ef3439be894 P ffd32ada67cd47368ab806773dadbd44: stopping tablet replica
I20260925 15:17:44.233911 31004 raft_consensus.cc:2229] T 9b0f9fd2e79b4e59949f5ef3439be894 P ffd32ada67cd47368ab806773dadbd44 [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:17:44.234620 31003 ts_tablet_manager.cc:1918] T 9b0f9fd2e79b4e59949f5ef3439be894 P 196922ce8c204035b9a27a502b2ffba7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:17:44.234686 31004 raft_consensus.cc:2258] T 9b0f9fd2e79b4e59949f5ef3439be894 P ffd32ada67cd47368ab806773dadbd44 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:17:44.234790 31003 log.cc:1196] T 9b0f9fd2e79b4e59949f5ef3439be894 P 196922ce8c204035b9a27a502b2ffba7: Deleting WAL directory at /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349162370577-12320-0/minicluster-data/ts-0/wal/wals/9b0f9fd2e79b4e59949f5ef3439be894
I20260925 15:17:44.235312 31003 ts_tablet_manager.cc:1939] T 9b0f9fd2e79b4e59949f5ef3439be894 P 196922ce8c204035b9a27a502b2ffba7: Deleting consensus metadata
I20260925 15:17:44.235836 31002 ts_tablet_manager.cc:1918] T 9b0f9fd2e79b4e59949f5ef3439be894 P 444292353eff41dc8a82548160c067b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:17:44.236074 31002 log.cc:1196] T 9b0f9fd2e79b4e59949f5ef3439be894 P 444292353eff41dc8a82548160c067b1: Deleting WAL directory at /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349162370577-12320-0/minicluster-data/ts-2/wal/wals/9b0f9fd2e79b4e59949f5ef3439be894
I20260925 15:17:44.236713 31002 ts_tablet_manager.cc:1939] T 9b0f9fd2e79b4e59949f5ef3439be894 P 444292353eff41dc8a82548160c067b1: Deleting consensus metadata
I20260925 15:17:44.237154 30491 catalog_manager.cc:4841] TS 196922ce8c204035b9a27a502b2ffba7 (127.12.8.1:40591): tablet 9b0f9fd2e79b4e59949f5ef3439be894 (table test-table [id=6efe80bbb4f74f009fc6c0afe7cdf7a7]) successfully deleted
I20260925 15:17:44.237452 31004 ts_tablet_manager.cc:1905] T 9b0f9fd2e79b4e59949f5ef3439be894 P ffd32ada67cd47368ab806773dadbd44: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:17:44.238564 30488 catalog_manager.cc:4841] TS 444292353eff41dc8a82548160c067b1 (127.12.8.3:45215): tablet 9b0f9fd2e79b4e59949f5ef3439be894 (table test-table [id=6efe80bbb4f74f009fc6c0afe7cdf7a7]) successfully deleted
Sep 25 15:17:44 dist-test-slave-p468 krb5kdc[30459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349461, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.1@KRBTEST.COM
I20260925 15:17:44.244830 31004 ts_tablet_manager.cc:1918] T 9b0f9fd2e79b4e59949f5ef3439be894 P ffd32ada67cd47368ab806773dadbd44: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:17:44.245029 31004 log.cc:1196] T 9b0f9fd2e79b4e59949f5ef3439be894 P ffd32ada67cd47368ab806773dadbd44: Deleting WAL directory at /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349162370577-12320-0/minicluster-data/ts-1/wal/wals/9b0f9fd2e79b4e59949f5ef3439be894
I20260925 15:17:44.245497 31004 ts_tablet_manager.cc:1939] T 9b0f9fd2e79b4e59949f5ef3439be894 P ffd32ada67cd47368ab806773dadbd44: Deleting consensus metadata
I20260925 15:17:44.247082 30491 catalog_manager.cc:4841] TS ffd32ada67cd47368ab806773dadbd44 (127.12.8.2:43573): tablet 9b0f9fd2e79b4e59949f5ef3439be894 (table test-table [id=6efe80bbb4f74f009fc6c0afe7cdf7a7]) successfully deleted
I20260925 15:17:44.251047 30973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:44.213546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34050 (local address 127.12.8.1:40591)
0925 15:17:44.213775 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:44.213784 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:44.229292 (+ 15508us) server_negotiation.cc:407] Connection header received
0925 15:17:44.229795 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:44.229803 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:44.229896 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:44.229968 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:17:44.232179 (+ 2211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:44.233360 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:44.237636 (+ 4276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:44.238316 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:44.238416 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:44.244141 (+ 5725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:17:44.244155 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:17:44.244173 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:17:44.244215 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:17:44.248106 (+ 3891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:17:44.248773 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:17:44.248783 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:17:44.248787 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:17:44.248946 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:17:44.249333 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:17:44.249338 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:17:44.249342 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:17:44.249785 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:17:44.250111 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:44.250399 (+ 288us) server_negotiation.cc:299] Negotiation successful
0925 15:17:44.250628 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 15:17:44.251727 30688 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:17:44.357030 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 30565
I20260925 15:17:44.381126 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 30699
Sep 25 15:17:44 dist-test-slave-p468 krb5kdc[30459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349461, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.2@KRBTEST.COM
I20260925 15:17:44.398720 30977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:44.381922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37480 (local address 127.12.8.2:43573)
0925 15:17:44.382105 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:44.382109 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:44.382165 (+ 56us) server_negotiation.cc:407] Connection header received
0925 15:17:44.382404 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:44.382408 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:44.382473 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:44.382559 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:17:44.384245 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:44.385285 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:44.387155 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:44.387563 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:44.387660 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:44.392571 (+ 4911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:17:44.392582 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:17:44.392597 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:17:44.392633 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:17:44.395756 (+ 3123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:17:44.396451 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:17:44.396459 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:17:44.396463 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:17:44.396618 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:17:44.396993 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:17:44.396999 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:17:44.397002 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:17:44.397385 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:17:44.397697 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:44.398062 (+ 365us) server_negotiation.cc:299] Negotiation successful
0925 15:17:44.398347 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 15:17:44.399451 30822 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:17:44.513448 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 30699
I20260925 15:17:44.534425 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 30832
Sep 25 15:17:44 dist-test-slave-p468 krb5kdc[30459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349461, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.3@KRBTEST.COM
I20260925 15:17:44.549786 30975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:44.535241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55518 (local address 127.12.8.3:45215)
0925 15:17:44.535430 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:44.535437 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:44.535472 (+ 35us) server_negotiation.cc:407] Connection header received
0925 15:17:44.535694 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:44.535704 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:44.535781 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:44.535889 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:17:44.537216 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:44.537955 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:44.539335 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:44.539753 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:44.539890 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:44.544454 (+ 4564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:17:44.544467 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:17:44.544482 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:17:44.544520 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:17:44.547422 (+ 2902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:17:44.547980 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:17:44.547986 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:17:44.547989 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:17:44.548107 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:17:44.548407 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:17:44.548410 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:17:44.548412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:17:44.548747 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:17:44.548954 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:44.549237 (+ 283us) server_negotiation.cc:299] Negotiation successful
0925 15:17:44.549387 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 15:17:44.550449 30955 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:17:44.640347 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 30832
I20260925 15:17:44.656934 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 30472
I20260925 15:17:44.658176 30534 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:17:44.768476 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 30472
2026-09-25T15:17:44Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4233 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-25T15:17:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:17:44Z Disabled control of system clock
I20260925 15:17:44.842650 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349162370577-12320-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349162370577-12320-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349162370577-12320-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.62:36117
--webserver_interface=127.12.8.62
--webserver_port=0
--builtin_ntp_servers=127.12.8.20:41383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.62:36117
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 15:17:45.162623 31013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:17:45.162889 31013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:17:45.163089 31013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:17:45.170146 31013 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:17:45.170315 31013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:17:45.170384 31013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:17:45.170490 31013 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:17:45.170562 31013 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:17:45.179586 31013 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:41383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349162370577-12320-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349162370577-12320-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.8.62:36117
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.62:36117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.8.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349162370577-12320-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:17:45.180830 31013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:17:45.182577 31013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:17:45.194258 31018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:17:45.194260 31019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:17:45.194325 31021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:17:45.195425 31013 server_base.cc:1034] running on GCE node
I20260925 15:17:45.196138 31013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:17:45.197423 31013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:17:45.198659 31013 hybrid_clock.cc:648] HybridClock initialized: now 1790349465198616 us; error 60 us; skew 500 ppm
I20260925 15:17:45.199016 31013 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:17:45.201189 31013 webserver.cc:460] Webserver started at http://127.12.8.62:43393/ using document root <none> and password file <none>
I20260925 15:17:45.201853 31013 fs_manager.cc:359] Metadata directory not provided
I20260925 15:17:45.201987 31013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:17:45.202365 31013 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:17:45.205248 31013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349162370577-12320-0/minicluster-data/master-0/data/instance:
uuid: "a59103211b834b6fb7e1c305b95739e5"
format_stamp: "Formatted at 2026-09-25 15:17:45 on dist-test-slave-p468"
server_key: "d1640a16ad1495af58724b6362d295f9"
server_key_iv: "a544b614e005d9bcc0624f9166f3e2dd"
server_key_version: "encryptionkey@0"
I20260925 15:17:45.206063 31013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349162370577-12320-0/minicluster-data/master-0/wal/instance:
uuid: "a59103211b834b6fb7e1c305b95739e5"
format_stamp: "Formatted at 2026-09-25 15:17:45 on dist-test-slave-p468"
server_key: "d1640a16ad1495af58724b6362d295f9"
server_key_iv: "a544b614e005d9bcc0624f9166f3e2dd"
server_key_version: "encryptionkey@0"
I20260925 15:17:45.210965 31013 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 15:17:45.214309 31027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:17:45.215584 31013 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 15:17:45.215768 31013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349162370577-12320-0/minicluster-data/master-0/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349162370577-12320-0/minicluster-data/master-0/wal
uuid: "a59103211b834b6fb7e1c305b95739e5"
format_stamp: "Formatted at 2026-09-25 15:17:45 on dist-test-slave-p468"
server_key: "d1640a16ad1495af58724b6362d295f9"
server_key_iv: "a544b614e005d9bcc0624f9166f3e2dd"
server_key_version: "encryptionkey@0"
I20260925 15:17:45.215975 31013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349162370577-12320-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349162370577-12320-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349162370577-12320-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:17:45.249094 31013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:17:45.249984 31013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:17:45.250352 31013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:17:45.268998 31013 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.62:36117
I20260925 15:17:45.269012 31078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.62:36117 every 8 connection(s)
I20260925 15:17:45.270466 31013 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
I20260925 15:17:45.274258 31079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:17:45.274466 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 31013
I20260925 15:17:45.274668 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349162370577-12320-0/minicluster-data/master-0/wal/instance
I20260925 15:17:45.275105 12320 external_mini_cluster.cc:1428] Setting key fb4e203c873ebf857258614948f8bfd3
I20260925 15:17:45.286513 31079 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a59103211b834b6fb7e1c305b95739e5: Bootstrap starting.
I20260925 15:17:45.288887 31082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:45.276969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59958 (local address 127.12.8.62:36117)
0925 15:17:45.278018 (+ 1049us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:45.278028 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:45.278067 (+ 39us) server_negotiation.cc:407] Connection header received
0925 15:17:45.280517 (+ 2450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:45.280538 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:45.281032 (+ 494us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:45.281451 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:17:45.281983 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:45.283280 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:45.284650 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:45.285293 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:45.285490 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:45.286748 (+ 1258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:17:45.286756 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:17:45.286769 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:17:45.287274 (+ 505us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:17:45.287346 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:17:45.287366 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:17:45.287494 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:45.287721 (+ 227us) server_negotiation.cc:299] Negotiation successful
0925 15:17:45.288028 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":624,"thread_start_us":399,"threads_started":1}
I20260925 15:17:45.289911 31079 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a59103211b834b6fb7e1c305b95739e5: Neither blocks nor log segments found. Creating new log.
I20260925 15:17:45.291023 31079 log.cc:824] T 00000000000000000000000000000000 P a59103211b834b6fb7e1c305b95739e5: Log is configured to *not* fsync() on all Append() calls
I20260925 15:17:45.293672 31079 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a59103211b834b6fb7e1c305b95739e5: No bootstrap required, opened a new log
I20260925 15:17:45.301072 31079 raft_consensus.cc:348] T 00000000000000000000000000000000 P a59103211b834b6fb7e1c305b95739e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a59103211b834b6fb7e1c305b95739e5" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 36117 } }
I20260925 15:17:45.301435 31079 raft_consensus.cc:374] T 00000000000000000000000000000000 P a59103211b834b6fb7e1c305b95739e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:17:45.301564 31079 raft_consensus.cc:729] T 00000000000000000000000000000000 P a59103211b834b6fb7e1c305b95739e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a59103211b834b6fb7e1c305b95739e5, State: Initialized, Role: FOLLOWER
I20260925 15:17:45.302119 31079 consensus_queue.cc:260] T 00000000000000000000000000000000 P a59103211b834b6fb7e1c305b95739e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a59103211b834b6fb7e1c305b95739e5" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 36117 } }
I20260925 15:17:45.302430 31079 raft_consensus.cc:388] T 00000000000000000000000000000000 P a59103211b834b6fb7e1c305b95739e5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:17:45.302582 31079 raft_consensus.cc:482] T 00000000000000000000000000000000 P a59103211b834b6fb7e1c305b95739e5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:17:45.302763 31079 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a59103211b834b6fb7e1c305b95739e5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:17:45.305303 31079 raft_consensus.cc:504] T 00000000000000000000000000000000 P a59103211b834b6fb7e1c305b95739e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a59103211b834b6fb7e1c305b95739e5" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 36117 } }
I20260925 15:17:45.305820 31079 leader_election.cc:302] T 00000000000000000000000000000000 P a59103211b834b6fb7e1c305b95739e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a59103211b834b6fb7e1c305b95739e5; no voters:
I20260925 15:17:45.306432 31079 leader_election.cc:288] T 00000000000000000000000000000000 P a59103211b834b6fb7e1c305b95739e5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:17:45.306571 31084 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a59103211b834b6fb7e1c305b95739e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:17:45.307274 31084 raft_consensus.cc:686] T 00000000000000000000000000000000 P a59103211b834b6fb7e1c305b95739e5 [term 1 LEADER]: Becoming Leader. State: Replica: a59103211b834b6fb7e1c305b95739e5, State: Running, Role: LEADER
I20260925 15:17:45.307754 31084 consensus_queue.cc:237] T 00000000000000000000000000000000 P a59103211b834b6fb7e1c305b95739e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a59103211b834b6fb7e1c305b95739e5" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 36117 } }
I20260925 15:17:45.308559 31079 sys_catalog.cc:564] T 00000000000000000000000000000000 P a59103211b834b6fb7e1c305b95739e5 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:17:45.310482 31085 sys_catalog.cc:455] T 00000000000000000000000000000000 P a59103211b834b6fb7e1c305b95739e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a59103211b834b6fb7e1c305b95739e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a59103211b834b6fb7e1c305b95739e5" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 36117 } } }
I20260925 15:17:45.310940 31085 sys_catalog.cc:458] T 00000000000000000000000000000000 P a59103211b834b6fb7e1c305b95739e5 [sys.catalog]: This master's current role is: LEADER
I20260925 15:17:45.310938 31086 sys_catalog.cc:455] T 00000000000000000000000000000000 P a59103211b834b6fb7e1c305b95739e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a59103211b834b6fb7e1c305b95739e5. Latest consensus state: current_term: 1 leader_uuid: "a59103211b834b6fb7e1c305b95739e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a59103211b834b6fb7e1c305b95739e5" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 36117 } } }
I20260925 15:17:45.311216 31086 sys_catalog.cc:458] T 00000000000000000000000000000000 P a59103211b834b6fb7e1c305b95739e5 [sys.catalog]: This master's current role is: LEADER
I20260925 15:17:45.313771 31092 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:17:45.318609 31092 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:17:45.326918 31092 catalog_manager.cc:1347] Generated new cluster ID: 8c3f65df94024d2a91bd8d18844fab1c
I20260925 15:17:45.327041 31092 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:17:45.337200 31092 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:17:45.338912 31092 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:17:45.346884 31092 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a59103211b834b6fb7e1c305b95739e5: Generated new TSK 0
I20260925 15:17:45.347671 31092 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:17:45.359403 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349162370577-12320-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349162370577-12320-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349162370577-12320-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349162370577-12320-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.1:0
--local_ip_for_outbound_sockets=127.12.8.1
--webserver_interface=127.12.8.1
--webserver_port=0
--tserver_master_addrs=127.12.8.62:36117
--builtin_ntp_servers=127.12.8.20:41383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 15:17:45.703161 31103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:17:45.703485 31103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:17:45.703652 31103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:17:45.711248 31103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:17:45.711375 31103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:17:45.711551 31103 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.1
I20260925 15:17:45.721040 31103 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:41383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349162370577-12320-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349162370577-12320-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349162370577-12320-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.8.1
--webserver_port=0
--tserver_master_addrs=127.12.8.62:36117
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.1
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349162370577-12320-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:17:45.722380 31103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:17:45.724148 31103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:17:45.737005 31111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:17:45.737027 31108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:17:45.737216 31109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:17:45.737813 31103 server_base.cc:1034] running on GCE node
I20260925 15:17:45.738425 31103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:17:45.739264 31103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:17:45.740456 31103 hybrid_clock.cc:648] HybridClock initialized: now 1790349465740393 us; error 56 us; skew 500 ppm
I20260925 15:17:45.740777 31103 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:17:45.742991 31103 webserver.cc:460] Webserver started at http://127.12.8.1:41711/ using document root <none> and password file <none>
I20260925 15:17:45.743649 31103 fs_manager.cc:359] Metadata directory not provided
I20260925 15:17:45.743767 31103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:17:45.744087 31103 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:17:45.746917 31103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349162370577-12320-0/minicluster-data/ts-0/data/instance:
uuid: "a4df706654f04cf9a8f0bec2678fc4fe"
format_stamp: "Formatted at 2026-09-25 15:17:45 on dist-test-slave-p468"
server_key: "ae63fcc0e81e02363b88a6bfb59ddaf1"
server_key_iv: "3a1ea20bdd07dda4484f040e978bb9e4"
server_key_version: "encryptionkey@0"
I20260925 15:17:45.747681 31103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349162370577-12320-0/minicluster-data/ts-0/wal/instance:
uuid: "a4df706654f04cf9a8f0bec2678fc4fe"
format_stamp: "Formatted at 2026-09-25 15:17:45 on dist-test-slave-p468"
server_key: "ae63fcc0e81e02363b88a6bfb59ddaf1"
server_key_iv: "3a1ea20bdd07dda4484f040e978bb9e4"
server_key_version: "encryptionkey@0"
I20260925 15:17:45.752671 31103 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 15:17:45.756140 31117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:17:45.757393 31103 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 15:17:45.757566 31103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349162370577-12320-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349162370577-12320-0/minicluster-data/ts-0/wal
uuid: "a4df706654f04cf9a8f0bec2678fc4fe"
format_stamp: "Formatted at 2026-09-25 15:17:45 on dist-test-slave-p468"
server_key: "ae63fcc0e81e02363b88a6bfb59ddaf1"
server_key_iv: "3a1ea20bdd07dda4484f040e978bb9e4"
server_key_version: "encryptionkey@0"
I20260925 15:17:45.757781 31103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349162370577-12320-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349162370577-12320-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349162370577-12320-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:17:45.793977 31103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:17:45.794920 31103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:17:45.795212 31103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:17:45.796185 31103 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:17:45.797672 31103 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:17:45.797756 31103 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:17:45.797906 31103 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:17:45.797988 31103 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:17:45.831831 31103 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.1:33233
I20260925 15:17:45.831928 31229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.1:33233 every 8 connection(s)
I20260925 15:17:45.833112 31103 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349162370577-12320-0/minicluster-data/ts-0/data/info.pb
I20260925 15:17:45.833583 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 31103
I20260925 15:17:45.833791 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349162370577-12320-0/minicluster-data/ts-0/wal/instance
I20260925 15:17:45.834491 12320 external_mini_cluster.cc:1428] Setting key 8449d6eac234281c11a28c959fb7f0db
I20260925 15:17:45.837512 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349162370577-12320-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349162370577-12320-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349162370577-12320-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349162370577-12320-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.2:0
--local_ip_for_outbound_sockets=127.12.8.2
--webserver_interface=127.12.8.2
--webserver_port=0
--tserver_master_addrs=127.12.8.62:36117
--builtin_ntp_servers=127.12.8.20:41383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 15:17:45.848178 31232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:45.836063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.1:46111 (local address 127.12.8.62:36117)
0925 15:17:45.836566 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:45.836572 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:45.838213 (+ 1641us) server_negotiation.cc:407] Connection header received
0925 15:17:45.840140 (+ 1927us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:45.840149 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:45.840243 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:45.840375 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:17:45.841464 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:45.842798 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:45.844471 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:45.845299 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:45.845454 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:45.847138 (+ 1684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:17:45.847147 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:17:45.847155 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:17:45.847180 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:17:45.847230 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:17:45.847249 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:17:45.847347 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:45.847607 (+ 260us) server_negotiation.cc:299] Negotiation successful
0925 15:17:45.847719 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":252,"threads_started":1}
I20260925 15:17:45.849184 31233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:45.836873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:36117 (local address 127.12.8.1:46111)
0925 15:17:45.838014 (+ 1141us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:45.838061 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 15:17:45.839852 (+ 1791us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:17:45.840582 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:17:45.840590 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:17:45.840681 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:17:45.841280 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:45.841297 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:45.842970 (+ 1673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:45.842974 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:45.844308 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:45.844321 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:45.845544 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:45.845553 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:45.845905 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:45.846750 (+ 845us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:17:45.846753 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:17:45.846794 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:17:45.846921 (+ 127us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:17:45.847382 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:17:45.847391 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:17:45.847398 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 15:17:45.847561 (+ 163us) client_negotiation.cc:240] Negotiation successful
0925 15:17:45.847860 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":390,"threads_started":1}
I20260925 15:17:45.850843 31230 heartbeater.cc:347] Connected to a master server at 127.12.8.62:36117
I20260925 15:17:45.851163 31230 heartbeater.cc:464] Registering TS with master...
I20260925 15:17:45.851909 31230 heartbeater.cc:511] Master 127.12.8.62:36117 requested a full tablet report, sending...
I20260925 15:17:45.854065 31044 ts_manager.cc:194] Registered new tserver with Master: a4df706654f04cf9a8f0bec2678fc4fe (127.12.8.1:33233)
I20260925 15:17:45.855710 31044 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.8.1:46111
W20260925 15:17:46.164079 31234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:17:46.164332 31234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:17:46.164471 31234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:17:46.171492 31234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:17:46.171581 31234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:17:46.171738 31234 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.2
I20260925 15:17:46.181130 31234 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:41383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349162370577-12320-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349162370577-12320-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349162370577-12320-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.8.2
--webserver_port=0
--tserver_master_addrs=127.12.8.62:36117
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.2
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349162370577-12320-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:17:46.182482 31234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:17:46.184157 31234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:17:46.196983 31240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:17:46.197118 31239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:17:46.197585 31242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:17:46.198120 31234 server_base.cc:1034] running on GCE node
I20260925 15:17:46.198729 31234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:17:46.199661 31234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:17:46.200879 31234 hybrid_clock.cc:648] HybridClock initialized: now 1790349466200848 us; error 40 us; skew 500 ppm
I20260925 15:17:46.201208 31234 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:17:46.203490 31234 webserver.cc:460] Webserver started at http://127.12.8.2:38913/ using document root <none> and password file <none>
I20260925 15:17:46.204130 31234 fs_manager.cc:359] Metadata directory not provided
I20260925 15:17:46.204243 31234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:17:46.204569 31234 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:17:46.207432 31234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349162370577-12320-0/minicluster-data/ts-1/data/instance:
uuid: "07cec26e186f4f4689355c14a130e769"
format_stamp: "Formatted at 2026-09-25 15:17:46 on dist-test-slave-p468"
server_key: "7694a238651b0c25b32747c44462c50b"
server_key_iv: "9e1f57f6d3276fca1e16b60d5d80f2bd"
server_key_version: "encryptionkey@0"
I20260925 15:17:46.208237 31234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349162370577-12320-0/minicluster-data/ts-1/wal/instance:
uuid: "07cec26e186f4f4689355c14a130e769"
format_stamp: "Formatted at 2026-09-25 15:17:46 on dist-test-slave-p468"
server_key: "7694a238651b0c25b32747c44462c50b"
server_key_iv: "9e1f57f6d3276fca1e16b60d5d80f2bd"
server_key_version: "encryptionkey@0"
I20260925 15:17:46.213251 31234 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260925 15:17:46.216675 31248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:17:46.217941 31234 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 15:17:46.218117 31234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349162370577-12320-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349162370577-12320-0/minicluster-data/ts-1/wal
uuid: "07cec26e186f4f4689355c14a130e769"
format_stamp: "Formatted at 2026-09-25 15:17:46 on dist-test-slave-p468"
server_key: "7694a238651b0c25b32747c44462c50b"
server_key_iv: "9e1f57f6d3276fca1e16b60d5d80f2bd"
server_key_version: "encryptionkey@0"
I20260925 15:17:46.218393 31234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349162370577-12320-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349162370577-12320-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349162370577-12320-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:17:46.234691 31234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:17:46.235468 31234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:17:46.235767 31234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:17:46.236704 31234 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:17:46.238157 31234 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:17:46.238354 31234 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:17:46.238436 31234 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:17:46.238494 31234 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:17:46.272802 31234 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.2:34365
I20260925 15:17:46.272855 31360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.2:34365 every 8 connection(s)
I20260925 15:17:46.274338 31234 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349162370577-12320-0/minicluster-data/ts-1/data/info.pb
I20260925 15:17:46.284325 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 31234
I20260925 15:17:46.284528 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349162370577-12320-0/minicluster-data/ts-1/wal/instance
I20260925 15:17:46.285084 12320 external_mini_cluster.cc:1428] Setting key 5cbe88124f31260f990d6dee6e48ef21
I20260925 15:17:46.288489 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349162370577-12320-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349162370577-12320-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349162370577-12320-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349162370577-12320-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.3:0
--local_ip_for_outbound_sockets=127.12.8.3
--webserver_interface=127.12.8.3
--webserver_port=0
--tserver_master_addrs=127.12.8.62:36117
--builtin_ntp_servers=127.12.8.20:41383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 15:17:46.288591 31232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:46.277010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.2:38437 (local address 127.12.8.62:36117)
0925 15:17:46.277239 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:46.277245 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:46.278848 (+ 1603us) server_negotiation.cc:407] Connection header received
0925 15:17:46.280568 (+ 1720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:46.280575 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:46.280656 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:46.280770 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:17:46.281800 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:46.282987 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:46.284949 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:46.285581 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:46.285702 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:46.287746 (+ 2044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:17:46.287753 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:17:46.287758 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:17:46.287780 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:17:46.287821 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:17:46.287836 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:17:46.287937 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:46.288177 (+ 240us) server_negotiation.cc:299] Negotiation successful
0925 15:17:46.288288 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260925 15:17:46.289678 31363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:46.277602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:36117 (local address 127.12.8.2:38437)
0925 15:17:46.278674 (+ 1072us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:46.278717 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 15:17:46.280291 (+ 1574us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:17:46.280985 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:17:46.280992 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:17:46.281073 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:17:46.281630 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:46.281647 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:46.283148 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:46.283152 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:46.284780 (+ 1628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:46.284795 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:46.285833 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:46.285842 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:46.286382 (+ 540us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:46.287307 (+ 925us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:17:46.287311 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:17:46.287425 (+ 114us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:17:46.287527 (+ 102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:17:46.287949 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:17:46.287958 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:17:46.287965 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 15:17:46.288122 (+ 157us) client_negotiation.cc:240] Negotiation successful
0925 15:17:46.288486 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":349,"threads_started":1}
I20260925 15:17:46.291234 31361 heartbeater.cc:347] Connected to a master server at 127.12.8.62:36117
I20260925 15:17:46.291616 31361 heartbeater.cc:464] Registering TS with master...
I20260925 15:17:46.292375 31361 heartbeater.cc:511] Master 127.12.8.62:36117 requested a full tablet report, sending...
I20260925 15:17:46.293757 31044 ts_manager.cc:194] Registered new tserver with Master: 07cec26e186f4f4689355c14a130e769 (127.12.8.2:34365)
I20260925 15:17:46.294648 31044 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.8.2:38437
W20260925 15:17:46.612115 31364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:17:46.612414 31364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:17:46.612581 31364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:17:46.619782 31364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:17:46.619866 31364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:17:46.620077 31364 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.3
I20260925 15:17:46.628931 31364 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:41383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349162370577-12320-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349162370577-12320-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349162370577-12320-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.8.3
--webserver_port=0
--tserver_master_addrs=127.12.8.62:36117
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.3
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349162370577-12320-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:17:46.630254 31364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:17:46.631958 31364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:17:46.644464 31369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:17:46.644557 31370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:17:46.644986 31372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:17:46.645421 31364 server_base.cc:1034] running on GCE node
I20260925 15:17:46.646054 31364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:17:46.646967 31364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:17:46.648183 31364 hybrid_clock.cc:648] HybridClock initialized: now 1790349466648149 us; error 29 us; skew 500 ppm
I20260925 15:17:46.648518 31364 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:17:46.650846 31364 webserver.cc:460] Webserver started at http://127.12.8.3:46053/ using document root <none> and password file <none>
I20260925 15:17:46.651580 31364 fs_manager.cc:359] Metadata directory not provided
I20260925 15:17:46.651705 31364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:17:46.652019 31364 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:17:46.654953 31364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349162370577-12320-0/minicluster-data/ts-2/data/instance:
uuid: "0df545d14e794b4d9f231ff0e2a451b5"
format_stamp: "Formatted at 2026-09-25 15:17:46 on dist-test-slave-p468"
server_key: "fd4e1a33714b49be82678b9c6ff5dc50"
server_key_iv: "078e90192ae5b9d91802144cee9a1272"
server_key_version: "encryptionkey@0"
I20260925 15:17:46.655761 31364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349162370577-12320-0/minicluster-data/ts-2/wal/instance:
uuid: "0df545d14e794b4d9f231ff0e2a451b5"
format_stamp: "Formatted at 2026-09-25 15:17:46 on dist-test-slave-p468"
server_key: "fd4e1a33714b49be82678b9c6ff5dc50"
server_key_iv: "078e90192ae5b9d91802144cee9a1272"
server_key_version: "encryptionkey@0"
I20260925 15:17:46.660789 31364 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 15:17:46.664175 31378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:17:46.665660 31364 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 15:17:46.665875 31364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349162370577-12320-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349162370577-12320-0/minicluster-data/ts-2/wal
uuid: "0df545d14e794b4d9f231ff0e2a451b5"
format_stamp: "Formatted at 2026-09-25 15:17:46 on dist-test-slave-p468"
server_key: "fd4e1a33714b49be82678b9c6ff5dc50"
server_key_iv: "078e90192ae5b9d91802144cee9a1272"
server_key_version: "encryptionkey@0"
I20260925 15:17:46.666108 31364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349162370577-12320-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349162370577-12320-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349162370577-12320-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:17:46.690593 31364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:17:46.691497 31364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:17:46.691828 31364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:17:46.692830 31364 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:17:46.694355 31364 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:17:46.694451 31364 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:17:46.694523 31364 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:17:46.694578 31364 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:17:46.728947 31364 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.3:42825
I20260925 15:17:46.729046 31490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.3:42825 every 8 connection(s)
I20260925 15:17:46.730280 31364 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349162370577-12320-0/minicluster-data/ts-2/data/info.pb
I20260925 15:17:46.732942 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 31364
I20260925 15:17:46.733182 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349162370577-12320-0/minicluster-data/ts-2/wal/instance
I20260925 15:17:46.733726 12320 external_mini_cluster.cc:1428] Setting key d76430195b616394a84da1b645dff67a
I20260925 15:17:46.743496 31232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:46.733055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.3:37061 (local address 127.12.8.62:36117)
0925 15:17:46.733258 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:46.733264 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:46.734937 (+ 1673us) server_negotiation.cc:407] Connection header received
0925 15:17:46.737013 (+ 2076us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:46.737017 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:46.737091 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:46.737182 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:17:46.738347 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:46.739148 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:46.740685 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:46.741094 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:46.741192 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:46.742789 (+ 1597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:17:46.742793 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:17:46.742797 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:17:46.742814 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:17:46.742846 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:17:46.742858 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:17:46.742936 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:46.743182 (+ 246us) server_negotiation.cc:299] Negotiation successful
0925 15:17:46.743259 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 15:17:46.744539 31493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:46.733634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:36117 (local address 127.12.8.3:37061)
0925 15:17:46.734766 (+ 1132us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:46.734812 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 15:17:46.736753 (+ 1941us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:17:46.737395 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:17:46.737404 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:17:46.737495 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:17:46.738122 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:46.738139 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:46.739290 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:46.739293 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:46.740551 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:46.740563 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:46.741324 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:46.741377 (+ 53us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:46.741623 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:46.742454 (+ 831us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:17:46.742457 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:17:46.742498 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:17:46.742585 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:17:46.742974 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:17:46.742982 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:17:46.742989 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 15:17:46.743138 (+ 149us) client_negotiation.cc:240] Negotiation successful
0925 15:17:46.743441 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":368,"threads_started":1}
I20260925 15:17:46.745887 31491 heartbeater.cc:347] Connected to a master server at 127.12.8.62:36117
I20260925 15:17:46.746239 31491 heartbeater.cc:464] Registering TS with master...
I20260925 15:17:46.747014 31491 heartbeater.cc:511] Master 127.12.8.62:36117 requested a full tablet report, sending...
I20260925 15:17:46.748700 31044 ts_manager.cc:194] Registered new tserver with Master: 0df545d14e794b4d9f231ff0e2a451b5 (127.12.8.3:42825)
I20260925 15:17:46.749436 12320 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:17:46.749801 31044 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.8.3:37061
I20260925 15:17:46.760280 31232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:46.754063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59970 (local address 127.12.8.62:36117)
0925 15:17:46.754307 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:46.754315 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:46.754740 (+ 425us) server_negotiation.cc:407] Connection header received
0925 15:17:46.754960 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:46.754970 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:46.755030 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:46.755146 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:17:46.755645 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:46.756341 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:46.757709 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:46.758124 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:46.758325 (+ 201us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:46.759522 (+ 1197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:17:46.759526 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:17:46.759529 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:17:46.759547 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:17:46.759580 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:17:46.759591 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:17:46.759747 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:46.759947 (+ 200us) server_negotiation.cc:299] Negotiation successful
0925 15:17:46.760004 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 15:17:46.764704 31044 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59970:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:17:46.772171 31044 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:17:46.805980 31504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:46.784713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.1:33233 (local address 127.0.0.1:57106)
0925 15:17:46.797677 (+ 12964us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:46.797698 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 15:17:46.797796 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:17:46.798967 (+ 1171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:17:46.798972 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:17:46.799003 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:17:46.799418 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:46.799428 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:46.800846 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:46.800851 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:46.803425 (+ 2574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:46.803440 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:46.805000 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:46.805006 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:46.805196 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:46.805250 (+ 54us) client_negotiation.cc:769] Sending connection context
0925 15:17:46.805381 (+ 131us) client_negotiation.cc:240] Negotiation successful
0925 15:17:46.805568 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12816,"mutex_wait_us":1,"thread_start_us":12461,"threads_started":1}
I20260925 15:17:46.806087 31506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:46.784927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57106 (local address 127.12.8.1:33233)
0925 15:17:46.798263 (+ 13336us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:46.798272 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:46.798315 (+ 43us) server_negotiation.cc:407] Connection header received
0925 15:17:46.798460 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:46.798470 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:46.798739 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:46.798941 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:17:46.799616 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:46.800700 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:46.803645 (+ 2945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:46.804806 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:46.804969 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:46.805224 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:46.805571 (+ 347us) server_negotiation.cc:299] Negotiation successful
0925 15:17:46.805738 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13176,"thread_start_us":13093,"threads_started":1}
I20260925 15:17:46.806752 31505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:46.784851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37868 (local address 127.12.8.2:34365)
0925 15:17:46.797056 (+ 12205us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:46.797066 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:46.797921 (+ 855us) server_negotiation.cc:407] Connection header received
0925 15:17:46.798026 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:46.798037 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:46.798377 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:46.798579 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:17:46.799639 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:46.801077 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:46.803743 (+ 2666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:46.805094 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:46.805222 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:46.805445 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:46.806247 (+ 802us) server_negotiation.cc:299] Negotiation successful
0925 15:17:46.806417 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12013,"thread_start_us":12204,"threads_started":1}
I20260925 15:17:46.806097 31503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:46.784645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.2:34365 (local address 127.0.0.1:37868)
0925 15:17:46.797270 (+ 12625us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:46.797332 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0925 15:17:46.797566 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:17:46.798842 (+ 1276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:17:46.798849 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:17:46.798882 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:17:46.799414 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:46.799429 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:46.801232 (+ 1803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:46.801236 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:46.803550 (+ 2314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:46.803563 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:46.805374 (+ 1811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:46.805380 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:46.805541 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:46.805570 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 15:17:46.805647 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 15:17:46.805758 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12359,"mutex_wait_us":24,"thread_start_us":13078,"threads_started":1}
I20260925 15:17:46.806841 31507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:46.787574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.3:42825 (local address 127.0.0.1:55630)
0925 15:17:46.798888 (+ 11314us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:46.798906 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 15:17:46.799028 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:17:46.799946 (+ 918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:17:46.799953 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:17:46.799982 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:17:46.800321 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:46.800332 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:46.802091 (+ 1759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:46.802098 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:46.803873 (+ 1775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:46.803891 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:46.805941 (+ 2050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:46.805947 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:46.806175 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:46.806244 (+ 69us) client_negotiation.cc:769] Sending connection context
0925 15:17:46.806330 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 15:17:46.806451 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11203,"mutex_wait_us":28,"thread_start_us":11063,"threads_started":1}
I20260925 15:17:46.807794 31508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:46.788252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55630 (local address 127.12.8.3:42825)
0925 15:17:46.798479 (+ 10227us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:46.798489 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:46.799157 (+ 668us) server_negotiation.cc:407] Connection header received
0925 15:17:46.799269 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:46.799278 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:46.799498 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:46.799668 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:17:46.800522 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:46.801928 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:46.804082 (+ 2154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:46.805431 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:46.805557 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:46.805801 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:46.806429 (+ 628us) server_negotiation.cc:299] Negotiation successful
0925 15:17:46.807435 (+ 1006us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10027,"thread_start_us":9964,"threads_started":1}
I20260925 15:17:46.810592 31165 tablet_service.cc:1467] Processing CreateTablet for tablet 7330540f8b8d4f1aaa54099dd503af7b (DEFAULT_TABLE table=test-table [id=17c014947eba498ba68b27eff80269d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:17:46.810592 31296 tablet_service.cc:1467] Processing CreateTablet for tablet 7330540f8b8d4f1aaa54099dd503af7b (DEFAULT_TABLE table=test-table [id=17c014947eba498ba68b27eff80269d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:17:46.811041 31426 tablet_service.cc:1467] Processing CreateTablet for tablet 7330540f8b8d4f1aaa54099dd503af7b (DEFAULT_TABLE table=test-table [id=17c014947eba498ba68b27eff80269d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:17:46.811987 31296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7330540f8b8d4f1aaa54099dd503af7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:17:46.812002 31165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7330540f8b8d4f1aaa54099dd503af7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:17:46.812278 31426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7330540f8b8d4f1aaa54099dd503af7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:17:46.826493 31510 tablet_bootstrap.cc:492] T 7330540f8b8d4f1aaa54099dd503af7b P a4df706654f04cf9a8f0bec2678fc4fe: Bootstrap starting.
I20260925 15:17:46.828732 31509 tablet_bootstrap.cc:492] T 7330540f8b8d4f1aaa54099dd503af7b P 0df545d14e794b4d9f231ff0e2a451b5: Bootstrap starting.
I20260925 15:17:46.829545 31511 tablet_bootstrap.cc:492] T 7330540f8b8d4f1aaa54099dd503af7b P 07cec26e186f4f4689355c14a130e769: Bootstrap starting.
I20260925 15:17:46.831840 31510 tablet_bootstrap.cc:654] T 7330540f8b8d4f1aaa54099dd503af7b P a4df706654f04cf9a8f0bec2678fc4fe: Neither blocks nor log segments found. Creating new log.
I20260925 15:17:46.832212 31509 tablet_bootstrap.cc:654] T 7330540f8b8d4f1aaa54099dd503af7b P 0df545d14e794b4d9f231ff0e2a451b5: Neither blocks nor log segments found. Creating new log.
I20260925 15:17:46.832696 31511 tablet_bootstrap.cc:654] T 7330540f8b8d4f1aaa54099dd503af7b P 07cec26e186f4f4689355c14a130e769: Neither blocks nor log segments found. Creating new log.
I20260925 15:17:46.833204 31510 log.cc:824] T 7330540f8b8d4f1aaa54099dd503af7b P a4df706654f04cf9a8f0bec2678fc4fe: Log is configured to *not* fsync() on all Append() calls
I20260925 15:17:46.833257 31509 log.cc:824] T 7330540f8b8d4f1aaa54099dd503af7b P 0df545d14e794b4d9f231ff0e2a451b5: Log is configured to *not* fsync() on all Append() calls
I20260925 15:17:46.834467 31511 log.cc:824] T 7330540f8b8d4f1aaa54099dd503af7b P 07cec26e186f4f4689355c14a130e769: Log is configured to *not* fsync() on all Append() calls
I20260925 15:17:46.842329 31510 tablet_bootstrap.cc:492] T 7330540f8b8d4f1aaa54099dd503af7b P a4df706654f04cf9a8f0bec2678fc4fe: No bootstrap required, opened a new log
I20260925 15:17:46.842317 31509 tablet_bootstrap.cc:492] T 7330540f8b8d4f1aaa54099dd503af7b P 0df545d14e794b4d9f231ff0e2a451b5: No bootstrap required, opened a new log
I20260925 15:17:46.842618 31510 ts_tablet_manager.cc:1397] T 7330540f8b8d4f1aaa54099dd503af7b P a4df706654f04cf9a8f0bec2678fc4fe: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.008s
I20260925 15:17:46.842610 31509 ts_tablet_manager.cc:1397] T 7330540f8b8d4f1aaa54099dd503af7b P 0df545d14e794b4d9f231ff0e2a451b5: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.008s
I20260925 15:17:46.843513 31511 tablet_bootstrap.cc:492] T 7330540f8b8d4f1aaa54099dd503af7b P 07cec26e186f4f4689355c14a130e769: No bootstrap required, opened a new log
I20260925 15:17:46.843834 31511 ts_tablet_manager.cc:1397] T 7330540f8b8d4f1aaa54099dd503af7b P 07cec26e186f4f4689355c14a130e769: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.004s
I20260925 15:17:46.853303 31510 raft_consensus.cc:348] T 7330540f8b8d4f1aaa54099dd503af7b P a4df706654f04cf9a8f0bec2678fc4fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cec26e186f4f4689355c14a130e769" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 34365 } } peers { permanent_uuid: "a4df706654f04cf9a8f0bec2678fc4fe" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 33233 } } peers { permanent_uuid: "0df545d14e794b4d9f231ff0e2a451b5" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 42825 } }
I20260925 15:17:46.853266 31509 raft_consensus.cc:348] T 7330540f8b8d4f1aaa54099dd503af7b P 0df545d14e794b4d9f231ff0e2a451b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cec26e186f4f4689355c14a130e769" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 34365 } } peers { permanent_uuid: "a4df706654f04cf9a8f0bec2678fc4fe" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 33233 } } peers { permanent_uuid: "0df545d14e794b4d9f231ff0e2a451b5" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 42825 } }
I20260925 15:17:46.853720 31510 raft_consensus.cc:374] T 7330540f8b8d4f1aaa54099dd503af7b P a4df706654f04cf9a8f0bec2678fc4fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:17:46.853713 31509 raft_consensus.cc:374] T 7330540f8b8d4f1aaa54099dd503af7b P 0df545d14e794b4d9f231ff0e2a451b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:17:46.853822 31509 raft_consensus.cc:729] T 7330540f8b8d4f1aaa54099dd503af7b P 0df545d14e794b4d9f231ff0e2a451b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0df545d14e794b4d9f231ff0e2a451b5, State: Initialized, Role: FOLLOWER
I20260925 15:17:46.853884 31510 raft_consensus.cc:729] T 7330540f8b8d4f1aaa54099dd503af7b P a4df706654f04cf9a8f0bec2678fc4fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4df706654f04cf9a8f0bec2678fc4fe, State: Initialized, Role: FOLLOWER
I20260925 15:17:46.853679 31511 raft_consensus.cc:348] T 7330540f8b8d4f1aaa54099dd503af7b P 07cec26e186f4f4689355c14a130e769 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cec26e186f4f4689355c14a130e769" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 34365 } } peers { permanent_uuid: "a4df706654f04cf9a8f0bec2678fc4fe" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 33233 } } peers { permanent_uuid: "0df545d14e794b4d9f231ff0e2a451b5" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 42825 } }
I20260925 15:17:46.854101 31511 raft_consensus.cc:374] T 7330540f8b8d4f1aaa54099dd503af7b P 07cec26e186f4f4689355c14a130e769 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:17:46.854262 31511 raft_consensus.cc:729] T 7330540f8b8d4f1aaa54099dd503af7b P 07cec26e186f4f4689355c14a130e769 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07cec26e186f4f4689355c14a130e769, State: Initialized, Role: FOLLOWER
I20260925 15:17:46.854478 31510 consensus_queue.cc:260] T 7330540f8b8d4f1aaa54099dd503af7b P a4df706654f04cf9a8f0bec2678fc4fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cec26e186f4f4689355c14a130e769" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 34365 } } peers { permanent_uuid: "a4df706654f04cf9a8f0bec2678fc4fe" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 33233 } } peers { permanent_uuid: "0df545d14e794b4d9f231ff0e2a451b5" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 42825 } }
I20260925 15:17:46.854478 31509 consensus_queue.cc:260] T 7330540f8b8d4f1aaa54099dd503af7b P 0df545d14e794b4d9f231ff0e2a451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cec26e186f4f4689355c14a130e769" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 34365 } } peers { permanent_uuid: "a4df706654f04cf9a8f0bec2678fc4fe" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 33233 } } peers { permanent_uuid: "0df545d14e794b4d9f231ff0e2a451b5" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 42825 } }
I20260925 15:17:46.854902 31511 consensus_queue.cc:260] T 7330540f8b8d4f1aaa54099dd503af7b P 07cec26e186f4f4689355c14a130e769 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cec26e186f4f4689355c14a130e769" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 34365 } } peers { permanent_uuid: "a4df706654f04cf9a8f0bec2678fc4fe" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 33233 } } peers { permanent_uuid: "0df545d14e794b4d9f231ff0e2a451b5" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 42825 } }
I20260925 15:17:46.855681 31491 heartbeater.cc:503] Master 127.12.8.62:36117 was elected leader, sending a full tablet report...
I20260925 15:17:46.856504 31509 ts_tablet_manager.cc:1428] T 7330540f8b8d4f1aaa54099dd503af7b P 0df545d14e794b4d9f231ff0e2a451b5: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260925 15:17:46.857496 31361 heartbeater.cc:503] Master 127.12.8.62:36117 was elected leader, sending a full tablet report...
I20260925 15:17:46.858036 31230 heartbeater.cc:503] Master 127.12.8.62:36117 was elected leader, sending a full tablet report...
I20260925 15:17:46.858136 31511 ts_tablet_manager.cc:1428] T 7330540f8b8d4f1aaa54099dd503af7b P 07cec26e186f4f4689355c14a130e769: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260925 15:17:46.859606 31510 ts_tablet_manager.cc:1428] T 7330540f8b8d4f1aaa54099dd503af7b P a4df706654f04cf9a8f0bec2678fc4fe: Time spent starting tablet: real 0.017s user 0.012s sys 0.003s
W20260925 15:17:46.982056 31492 tablet.cc:2369] T 7330540f8b8d4f1aaa54099dd503af7b P 0df545d14e794b4d9f231ff0e2a451b5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:17:46.993813 31516 raft_consensus.cc:482] T 7330540f8b8d4f1aaa54099dd503af7b P a4df706654f04cf9a8f0bec2678fc4fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:17:46.994215 31516 raft_consensus.cc:504] T 7330540f8b8d4f1aaa54099dd503af7b P a4df706654f04cf9a8f0bec2678fc4fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cec26e186f4f4689355c14a130e769" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 34365 } } peers { permanent_uuid: "a4df706654f04cf9a8f0bec2678fc4fe" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 33233 } } peers { permanent_uuid: "0df545d14e794b4d9f231ff0e2a451b5" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 42825 } }
I20260925 15:17:46.996188 31516 leader_election.cc:288] T 7330540f8b8d4f1aaa54099dd503af7b P a4df706654f04cf9a8f0bec2678fc4fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 07cec26e186f4f4689355c14a130e769 (127.12.8.2:34365), 0df545d14e794b4d9f231ff0e2a451b5 (127.12.8.3:42825)
I20260925 15:17:47.003074 31508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:46.996709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.1:36195 (local address 127.12.8.3:42825)
0925 15:17:46.996969 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:46.996979 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:46.997759 (+ 780us) server_negotiation.cc:407] Connection header received
0925 15:17:46.997987 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:46.997992 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:46.998064 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:46.998164 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:17:46.998721 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:46.999436 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:47.001370 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:47.002147 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:47.002337 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:47.002438 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:47.002711 (+ 273us) server_negotiation.cc:299] Negotiation successful
0925 15:17:47.002833 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 15:17:47.003180 31518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:46.996487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.3:42825 (local address 127.12.8.1:36195)
0925 15:17:46.997627 (+ 1140us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:46.997645 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 15:17:46.997837 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:17:46.998181 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:17:46.998185 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:17:46.998233 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:17:46.998573 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:46.998582 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:46.999577 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:46.999584 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:47.001177 (+ 1593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:47.001188 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:47.002371 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:47.002377 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:47.002540 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:47.002570 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 15:17:47.002684 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 15:17:47.002816 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1006,"mutex_wait_us":83,"thread_start_us":788,"threads_started":1}
I20260925 15:17:47.003324 31505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:46.996600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.1:51241 (local address 127.12.8.2:34365)
0925 15:17:46.996835 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:46.996844 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:46.998070 (+ 1226us) server_negotiation.cc:407] Connection header received
0925 15:17:46.998317 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:46.998323 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:46.998425 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:46.998532 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:17:46.999090 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:46.999752 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:47.001582 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:47.002404 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:47.002554 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:47.002666 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:47.002943 (+ 277us) server_negotiation.cc:299] Negotiation successful
0925 15:17:47.003075 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 15:17:47.003726 31519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:46.996590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.2:34365 (local address 127.12.8.1:51241)
0925 15:17:46.997823 (+ 1233us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:46.997882 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 15:17:46.998136 (+ 254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:17:46.998563 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:17:46.998568 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:17:46.998591 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:17:46.998908 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:46.998916 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:46.999901 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:46.999906 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:47.001384 (+ 1478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:47.001396 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:47.002612 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:47.002618 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:47.002781 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:47.002803 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 15:17:47.002914 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 15:17:47.003032 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1113,"mutex_wait_us":42,"thread_start_us":853,"threads_started":1}
I20260925 15:17:47.004243 31446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7330540f8b8d4f1aaa54099dd503af7b" candidate_uuid: "a4df706654f04cf9a8f0bec2678fc4fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0df545d14e794b4d9f231ff0e2a451b5" is_pre_election: true
I20260925 15:17:47.004688 31316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7330540f8b8d4f1aaa54099dd503af7b" candidate_uuid: "a4df706654f04cf9a8f0bec2678fc4fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07cec26e186f4f4689355c14a130e769" is_pre_election: true
I20260925 15:17:47.004793 31446 raft_consensus.cc:2454] T 7330540f8b8d4f1aaa54099dd503af7b P 0df545d14e794b4d9f231ff0e2a451b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4df706654f04cf9a8f0bec2678fc4fe in term 0.
I20260925 15:17:47.005168 31316 raft_consensus.cc:2454] T 7330540f8b8d4f1aaa54099dd503af7b P 07cec26e186f4f4689355c14a130e769 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4df706654f04cf9a8f0bec2678fc4fe in term 0.
I20260925 15:17:47.005497 31121 leader_election.cc:302] T 7330540f8b8d4f1aaa54099dd503af7b P a4df706654f04cf9a8f0bec2678fc4fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0df545d14e794b4d9f231ff0e2a451b5, a4df706654f04cf9a8f0bec2678fc4fe; no voters:
I20260925 15:17:47.005939 31516 raft_consensus.cc:2781] T 7330540f8b8d4f1aaa54099dd503af7b P a4df706654f04cf9a8f0bec2678fc4fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:17:47.006114 31516 raft_consensus.cc:482] T 7330540f8b8d4f1aaa54099dd503af7b P a4df706654f04cf9a8f0bec2678fc4fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:17:47.006244 31516 raft_consensus.cc:3037] T 7330540f8b8d4f1aaa54099dd503af7b P a4df706654f04cf9a8f0bec2678fc4fe [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:17:47.008584 31516 raft_consensus.cc:504] T 7330540f8b8d4f1aaa54099dd503af7b P a4df706654f04cf9a8f0bec2678fc4fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cec26e186f4f4689355c14a130e769" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 34365 } } peers { permanent_uuid: "a4df706654f04cf9a8f0bec2678fc4fe" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 33233 } } peers { permanent_uuid: "0df545d14e794b4d9f231ff0e2a451b5" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 42825 } }
I20260925 15:17:47.009253 31516 leader_election.cc:288] T 7330540f8b8d4f1aaa54099dd503af7b P a4df706654f04cf9a8f0bec2678fc4fe [CANDIDATE]: Term 1 election: Requested vote from peers 07cec26e186f4f4689355c14a130e769 (127.12.8.2:34365), 0df545d14e794b4d9f231ff0e2a451b5 (127.12.8.3:42825)
I20260925 15:17:47.009701 31316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7330540f8b8d4f1aaa54099dd503af7b" candidate_uuid: "a4df706654f04cf9a8f0bec2678fc4fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07cec26e186f4f4689355c14a130e769"
I20260925 15:17:47.009716 31446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7330540f8b8d4f1aaa54099dd503af7b" candidate_uuid: "a4df706654f04cf9a8f0bec2678fc4fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0df545d14e794b4d9f231ff0e2a451b5"
I20260925 15:17:47.009896 31316 raft_consensus.cc:3037] T 7330540f8b8d4f1aaa54099dd503af7b P 07cec26e186f4f4689355c14a130e769 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:17:47.009912 31446 raft_consensus.cc:3037] T 7330540f8b8d4f1aaa54099dd503af7b P 0df545d14e794b4d9f231ff0e2a451b5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:17:47.012025 31446 raft_consensus.cc:2454] T 7330540f8b8d4f1aaa54099dd503af7b P 0df545d14e794b4d9f231ff0e2a451b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4df706654f04cf9a8f0bec2678fc4fe in term 1.
I20260925 15:17:47.012243 31316 raft_consensus.cc:2454] T 7330540f8b8d4f1aaa54099dd503af7b P 07cec26e186f4f4689355c14a130e769 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4df706654f04cf9a8f0bec2678fc4fe in term 1.
I20260925 15:17:47.012421 31121 leader_election.cc:302] T 7330540f8b8d4f1aaa54099dd503af7b P a4df706654f04cf9a8f0bec2678fc4fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0df545d14e794b4d9f231ff0e2a451b5, a4df706654f04cf9a8f0bec2678fc4fe; no voters:
I20260925 15:17:47.012712 31516 raft_consensus.cc:2781] T 7330540f8b8d4f1aaa54099dd503af7b P a4df706654f04cf9a8f0bec2678fc4fe [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:17:47.013164 31516 raft_consensus.cc:686] T 7330540f8b8d4f1aaa54099dd503af7b P a4df706654f04cf9a8f0bec2678fc4fe [term 1 LEADER]: Becoming Leader. State: Replica: a4df706654f04cf9a8f0bec2678fc4fe, State: Running, Role: LEADER
I20260925 15:17:47.013664 31516 consensus_queue.cc:237] T 7330540f8b8d4f1aaa54099dd503af7b P a4df706654f04cf9a8f0bec2678fc4fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cec26e186f4f4689355c14a130e769" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 34365 } } peers { permanent_uuid: "a4df706654f04cf9a8f0bec2678fc4fe" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 33233 } } peers { permanent_uuid: "0df545d14e794b4d9f231ff0e2a451b5" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 42825 } }
I20260925 15:17:47.018035 31044 catalog_manager.cc:5495] T 7330540f8b8d4f1aaa54099dd503af7b P a4df706654f04cf9a8f0bec2678fc4fe reported cstate change: term changed from 0 to 1, leader changed from <none> to a4df706654f04cf9a8f0bec2678fc4fe (127.12.8.1). New cstate: current_term: 1 leader_uuid: "a4df706654f04cf9a8f0bec2678fc4fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cec26e186f4f4689355c14a130e769" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 34365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4df706654f04cf9a8f0bec2678fc4fe" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 33233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0df545d14e794b4d9f231ff0e2a451b5" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 42825 } health_report { overall_health: UNKNOWN } } }
W20260925 15:17:47.026468 31362 tablet.cc:2369] T 7330540f8b8d4f1aaa54099dd503af7b P 07cec26e186f4f4689355c14a130e769: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:17:47.085659 31231 tablet.cc:2369] T 7330540f8b8d4f1aaa54099dd503af7b P a4df706654f04cf9a8f0bec2678fc4fe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:17:47.090001 31506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:47.084620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57110 (local address 127.12.8.1:33233)
0925 15:17:47.084838 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:47.084842 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:47.084860 (+ 18us) server_negotiation.cc:407] Connection header received
0925 15:17:47.084983 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:47.084986 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:47.085073 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:47.085195 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:17:47.085564 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:47.086403 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:47.087786 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:47.088209 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:47.088350 (+ 141us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:47.088933 (+ 583us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:17:47.089402 (+ 469us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:17:47.089593 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:47.089740 (+ 147us) server_negotiation.cc:299] Negotiation successful
0925 15:17:47.089784 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 15:17:47.097951 31316 raft_consensus.cc:1261] T 7330540f8b8d4f1aaa54099dd503af7b P 07cec26e186f4f4689355c14a130e769 [term 1 FOLLOWER]: Refusing update from remote peer a4df706654f04cf9a8f0bec2678fc4fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:17:47.097949 31446 raft_consensus.cc:1261] T 7330540f8b8d4f1aaa54099dd503af7b P 0df545d14e794b4d9f231ff0e2a451b5 [term 1 FOLLOWER]: Refusing update from remote peer a4df706654f04cf9a8f0bec2678fc4fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:17:47.099042 31520 consensus_queue.cc:1035] T 7330540f8b8d4f1aaa54099dd503af7b P a4df706654f04cf9a8f0bec2678fc4fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "07cec26e186f4f4689355c14a130e769" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 34365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:17:47.099391 31516 consensus_queue.cc:1035] T 7330540f8b8d4f1aaa54099dd503af7b P a4df706654f04cf9a8f0bec2678fc4fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "0df545d14e794b4d9f231ff0e2a451b5" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 42825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:17:47.118692 31523 mvcc.cc:205] Tried to move back new op lower bound from 7333271417222479872 to 7333271416892080128. Current Snapshot: MvccSnapshot[applied={T|T < 7333271417222479872 or (T in {7333271417222479872})}]
I20260925 15:17:47.124817 31044 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59970:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:17:47.125113 31044 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59970:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:17:47.128455 31044 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a59103211b834b6fb7e1c305b95739e5: Sending DeleteTablet for 3 replicas of tablet 7330540f8b8d4f1aaa54099dd503af7b
I20260925 15:17:47.129560 31296 tablet_service.cc:1514] Processing DeleteTablet for tablet 7330540f8b8d4f1aaa54099dd503af7b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:17:47 UTC) from {username='slave'} at 127.0.0.1:37868
I20260925 15:17:47.129606 31165 tablet_service.cc:1514] Processing DeleteTablet for tablet 7330540f8b8d4f1aaa54099dd503af7b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:17:47 UTC) from {username='slave'} at 127.0.0.1:57106
I20260925 15:17:47.129848 31426 tablet_service.cc:1514] Processing DeleteTablet for tablet 7330540f8b8d4f1aaa54099dd503af7b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:17:47 UTC) from {username='slave'} at 127.0.0.1:55630
I20260925 15:17:47.131767 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 31103
I20260925 15:17:47.139890 31533 tablet_replica.cc:330] T 7330540f8b8d4f1aaa54099dd503af7b P a4df706654f04cf9a8f0bec2678fc4fe: stopping tablet replica
I20260925 15:17:47.140403 31533 raft_consensus.cc:2229] T 7330540f8b8d4f1aaa54099dd503af7b P a4df706654f04cf9a8f0bec2678fc4fe [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:17:47.141115 31533 raft_consensus.cc:2258] T 7330540f8b8d4f1aaa54099dd503af7b P a4df706654f04cf9a8f0bec2678fc4fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:17:47.141906 31532 tablet_replica.cc:330] T 7330540f8b8d4f1aaa54099dd503af7b P 07cec26e186f4f4689355c14a130e769: stopping tablet replica
I20260925 15:17:47.142382 31532 raft_consensus.cc:2229] T 7330540f8b8d4f1aaa54099dd503af7b P 07cec26e186f4f4689355c14a130e769 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:17:47.142927 31532 raft_consensus.cc:2258] T 7330540f8b8d4f1aaa54099dd503af7b P 07cec26e186f4f4689355c14a130e769 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:17:47.145699 31533 ts_tablet_manager.cc:1905] T 7330540f8b8d4f1aaa54099dd503af7b P a4df706654f04cf9a8f0bec2678fc4fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:17:47.146453 31532 ts_tablet_manager.cc:1905] T 7330540f8b8d4f1aaa54099dd503af7b P 07cec26e186f4f4689355c14a130e769: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:17:47.153137 31533 ts_tablet_manager.cc:1918] T 7330540f8b8d4f1aaa54099dd503af7b P a4df706654f04cf9a8f0bec2678fc4fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:17:47.153375 31533 log.cc:1196] T 7330540f8b8d4f1aaa54099dd503af7b P a4df706654f04cf9a8f0bec2678fc4fe: Deleting WAL directory at /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349162370577-12320-0/minicluster-data/ts-0/wal/wals/7330540f8b8d4f1aaa54099dd503af7b
I20260925 15:17:47.154003 31533 ts_tablet_manager.cc:1939] T 7330540f8b8d4f1aaa54099dd503af7b P a4df706654f04cf9a8f0bec2678fc4fe: Deleting consensus metadata
I20260925 15:17:47.155857 31029 catalog_manager.cc:4841] TS a4df706654f04cf9a8f0bec2678fc4fe (127.12.8.1:33233): tablet 7330540f8b8d4f1aaa54099dd503af7b (table test-table [id=17c014947eba498ba68b27eff80269d9]) successfully deleted
I20260925 15:17:47.158715 31534 tablet_replica.cc:330] T 7330540f8b8d4f1aaa54099dd503af7b P 0df545d14e794b4d9f231ff0e2a451b5: stopping tablet replica
I20260925 15:17:47.159123 31534 raft_consensus.cc:2229] T 7330540f8b8d4f1aaa54099dd503af7b P 0df545d14e794b4d9f231ff0e2a451b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:17:47.159575 31534 raft_consensus.cc:2258] T 7330540f8b8d4f1aaa54099dd503af7b P 0df545d14e794b4d9f231ff0e2a451b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:17:47.160518 31532 ts_tablet_manager.cc:1918] T 7330540f8b8d4f1aaa54099dd503af7b P 07cec26e186f4f4689355c14a130e769: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:17:47.160842 31532 log.cc:1196] T 7330540f8b8d4f1aaa54099dd503af7b P 07cec26e186f4f4689355c14a130e769: Deleting WAL directory at /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349162370577-12320-0/minicluster-data/ts-1/wal/wals/7330540f8b8d4f1aaa54099dd503af7b
I20260925 15:17:47.161428 31532 ts_tablet_manager.cc:1939] T 7330540f8b8d4f1aaa54099dd503af7b P 07cec26e186f4f4689355c14a130e769: Deleting consensus metadata
I20260925 15:17:47.162777 31534 ts_tablet_manager.cc:1905] T 7330540f8b8d4f1aaa54099dd503af7b P 0df545d14e794b4d9f231ff0e2a451b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:17:47.163271 31030 catalog_manager.cc:4841] TS 07cec26e186f4f4689355c14a130e769 (127.12.8.2:34365): tablet 7330540f8b8d4f1aaa54099dd503af7b (table test-table [id=17c014947eba498ba68b27eff80269d9]) successfully deleted
I20260925 15:17:47.164269 31506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:47.133476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57122 (local address 127.12.8.1:33233)
0925 15:17:47.134106 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:47.134115 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:47.153109 (+ 18994us) server_negotiation.cc:407] Connection header received
0925 15:17:47.153383 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:47.153391 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:47.153491 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:47.153604 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:17:47.156788 (+ 3184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:47.157977 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:47.160063 (+ 2086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:47.160759 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:47.160860 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:47.162448 (+ 1588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:17:47.162459 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:17:47.162475 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:17:47.163259 (+ 784us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:17:47.163320 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:17:47.163344 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:17:47.163431 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:47.163745 (+ 314us) server_negotiation.cc:299] Negotiation successful
0925 15:17:47.163852 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451}
I20260925 15:17:47.165028 31225 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:17:47.170234 31534 ts_tablet_manager.cc:1918] T 7330540f8b8d4f1aaa54099dd503af7b P 0df545d14e794b4d9f231ff0e2a451b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:17:47.170404 31534 log.cc:1196] T 7330540f8b8d4f1aaa54099dd503af7b P 0df545d14e794b4d9f231ff0e2a451b5: Deleting WAL directory at /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349162370577-12320-0/minicluster-data/ts-2/wal/wals/7330540f8b8d4f1aaa54099dd503af7b
I20260925 15:17:47.170902 31534 ts_tablet_manager.cc:1939] T 7330540f8b8d4f1aaa54099dd503af7b P 0df545d14e794b4d9f231ff0e2a451b5: Deleting consensus metadata
I20260925 15:17:47.172564 31031 catalog_manager.cc:4841] TS 0df545d14e794b4d9f231ff0e2a451b5 (127.12.8.3:42825): tablet 7330540f8b8d4f1aaa54099dd503af7b (table test-table [id=17c014947eba498ba68b27eff80269d9]) successfully deleted
I20260925 15:17:47.264364 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 31103
I20260925 15:17:47.282378 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 31234
I20260925 15:17:47.289608 31505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:47.283119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37882 (local address 127.12.8.2:34365)
0925 15:17:47.283302 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:47.283309 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:47.283479 (+ 170us) server_negotiation.cc:407] Connection header received
0925 15:17:47.283699 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:47.283705 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:47.283807 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:47.283928 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:17:47.284317 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:47.285102 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:47.286572 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:47.286989 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:47.287129 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:47.288201 (+ 1072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:17:47.288209 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:17:47.288222 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:17:47.288794 (+ 572us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:17:47.288840 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:17:47.288863 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:17:47.288985 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:47.289132 (+ 147us) server_negotiation.cc:299] Negotiation successful
0925 15:17:47.289233 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 15:17:47.290341 31356 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:17:47.379498 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 31234
I20260925 15:17:47.396392 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 31364
I20260925 15:17:47.403680 31508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:47.397217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55646 (local address 127.12.8.3:42825)
0925 15:17:47.397401 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:47.397409 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:47.397490 (+ 81us) server_negotiation.cc:407] Connection header received
0925 15:17:47.397723 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:47.397727 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:47.397815 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:47.397929 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:17:47.398439 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:47.399150 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:47.400499 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:47.400913 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:47.401058 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:47.402338 (+ 1280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:17:47.402349 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:17:47.402364 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:17:47.402932 (+ 568us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:17:47.402979 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:17:47.402998 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:17:47.403136 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:47.403233 (+ 97us) server_negotiation.cc:299] Negotiation successful
0925 15:17:47.403342 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 15:17:47.404346 31486 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:17:47.491618 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 31364
I20260925 15:17:47.507793 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 31013
I20260925 15:17:47.508962 31074 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:17:47.616569 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 31013
2026-09-25T15:17:47Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2846 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349162370577-12320-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:17:47 dist-test-slave-p468 krb5kdc[31542](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:17:47 dist-test-slave-p468 krb5kdc[31542](info): set up 2 sockets
Sep 25 15:17:47 dist-test-slave-p468 krb5kdc[31542](info): commencing operation
krb5kdc: starting...
W20260925 15:17:48.714546 12320 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349162370577-12320-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349162370577-12320-0/krb5kdc/test-user.keytab.
Sep 25 15:17:48 dist-test-slave-p468 krb5kdc[31542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:17:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:17:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.8.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab.
Entry for principal kudu/127.12.8.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab.
I20260925 15:17:48.966205 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349162370577-12320-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349162370577-12320-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349162370577-12320-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.62:45791
--webserver_interface=127.12.8.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab
--principal=kudu/127.12.8.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:34103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.62:45791
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:17:49.286653 31555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:17:49.286913 31555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:17:49.287107 31555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:17:49.294020 31555 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:17:49.294128 31555 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:17:49.294272 31555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:17:49.294371 31555 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:17:49.294456 31555 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:17:49.303897 31555 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:34103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349162370577-12320-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349162370577-12320-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.8.62:45791
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab
--principal=kudu/127.12.8.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.62:45791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.8.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349162370577-12320-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:17:49.305178 31555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:17:49.306514 31555 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:17:49.318543 31560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:17:49.318567 31561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:17:49.318656 31563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:17:49.319458 31555 server_base.cc:1034] running on GCE node
I20260925 15:17:49.320228 31555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:17:49.321645 31555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:17:49.322875 31555 hybrid_clock.cc:648] HybridClock initialized: now 1790349469322829 us; error 69 us; skew 500 ppm
I20260925 15:17:49.323220 31555 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:17:49 dist-test-slave-p468 krb5kdc[31542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349469, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:17:49.332964 31555 init.cc:377] Logged in from keytab as kudu/127.12.8.62@KRBTEST.COM (short username kudu)
I20260925 15:17:49.334514 31555 webserver.cc:460] Webserver started at http://127.12.8.62:34331/ using document root <none> and password file <none>
I20260925 15:17:49.335150 31555 fs_manager.cc:359] Metadata directory not provided
I20260925 15:17:49.335248 31555 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:17:49.335494 31555 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:17:49.338377 31555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349162370577-12320-0/minicluster-data/master-0/data/instance:
uuid: "8d02d3aef7a64832ba10d91c18a30c41"
format_stamp: "Formatted at 2026-09-25 15:17:49 on dist-test-slave-p468"
server_key: "2ac55c50d3be2e919da2d0ac9911ab26"
server_key_iv: "3a8ed88491fa7c69b9e18d53dc12fc74"
server_key_version: "encryptionkey@0"
I20260925 15:17:49.339078 31555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349162370577-12320-0/minicluster-data/master-0/wal/instance:
uuid: "8d02d3aef7a64832ba10d91c18a30c41"
format_stamp: "Formatted at 2026-09-25 15:17:49 on dist-test-slave-p468"
server_key: "2ac55c50d3be2e919da2d0ac9911ab26"
server_key_iv: "3a8ed88491fa7c69b9e18d53dc12fc74"
server_key_version: "encryptionkey@0"
I20260925 15:17:49.343983 31555 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 15:17:49.347275 31570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:17:49.348541 31555 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:17:49.348733 31555 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349162370577-12320-0/minicluster-data/master-0/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349162370577-12320-0/minicluster-data/master-0/wal
uuid: "8d02d3aef7a64832ba10d91c18a30c41"
format_stamp: "Formatted at 2026-09-25 15:17:49 on dist-test-slave-p468"
server_key: "2ac55c50d3be2e919da2d0ac9911ab26"
server_key_iv: "3a8ed88491fa7c69b9e18d53dc12fc74"
server_key_version: "encryptionkey@0"
I20260925 15:17:49.348902 31555 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349162370577-12320-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349162370577-12320-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349162370577-12320-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:17:49.373757 31555 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:17:49.377820 31555 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:17:49.378123 31555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:17:49.396641 31555 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.62:45791
I20260925 15:17:49.396680 31621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.62:45791 every 8 connection(s)
I20260925 15:17:49.398026 31555 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
I20260925 15:17:49.399751 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 31555
I20260925 15:17:49.399941 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349162370577-12320-0/minicluster-data/master-0/wal/instance
I20260925 15:17:49.400497 12320 external_mini_cluster.cc:1428] Setting key 00ef767af99404bbb788fa86b33b810c
I20260925 15:17:49.402830 31622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 25 15:17:49 dist-test-slave-p468 krb5kdc[31542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
I20260925 15:17:49.416177 31622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d02d3aef7a64832ba10d91c18a30c41: Bootstrap starting.
I20260925 15:17:49.419410 31622 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d02d3aef7a64832ba10d91c18a30c41: Neither blocks nor log segments found. Creating new log.
I20260925 15:17:49.420732 31622 log.cc:824] T 00000000000000000000000000000000 P 8d02d3aef7a64832ba10d91c18a30c41: Log is configured to *not* fsync() on all Append() calls
I20260925 15:17:49.423763 31625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:49.402536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53454 (local address 127.12.8.62:45791)
0925 15:17:49.403324 (+ 788us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:49.403335 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:49.403374 (+ 39us) server_negotiation.cc:407] Connection header received
0925 15:17:49.405006 (+ 1632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:49.405026 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:49.405410 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:49.405786 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:17:49.407602 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:49.408933 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:49.410788 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:49.411464 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:49.411599 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:49.416377 (+ 4778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:17:49.416392 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:17:49.416406 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:17:49.416440 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:17:49.419501 (+ 3061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:17:49.420388 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:17:49.420394 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:17:49.420400 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:17:49.420577 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:17:49.421009 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:17:49.421012 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:17:49.421014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:17:49.421757 (+ 743us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:17:49.422094 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:49.422447 (+ 353us) server_negotiation.cc:299] Negotiation successful
0925 15:17:49.422813 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":376,"threads_started":1}
I20260925 15:17:49.423954 31622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d02d3aef7a64832ba10d91c18a30c41: No bootstrap required, opened a new log
I20260925 15:17:49.431355 31622 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d02d3aef7a64832ba10d91c18a30c41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d02d3aef7a64832ba10d91c18a30c41" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 45791 } }
I20260925 15:17:49.431723 31622 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d02d3aef7a64832ba10d91c18a30c41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:17:49.431838 31622 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d02d3aef7a64832ba10d91c18a30c41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d02d3aef7a64832ba10d91c18a30c41, State: Initialized, Role: FOLLOWER
I20260925 15:17:49.432391 31622 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d02d3aef7a64832ba10d91c18a30c41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d02d3aef7a64832ba10d91c18a30c41" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 45791 } }
I20260925 15:17:49.432639 31622 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d02d3aef7a64832ba10d91c18a30c41 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:17:49.432791 31622 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d02d3aef7a64832ba10d91c18a30c41 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:17:49.432964 31622 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d02d3aef7a64832ba10d91c18a30c41 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:17:49.435102 31622 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d02d3aef7a64832ba10d91c18a30c41 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d02d3aef7a64832ba10d91c18a30c41" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 45791 } }
I20260925 15:17:49.435570 31622 leader_election.cc:302] T 00000000000000000000000000000000 P 8d02d3aef7a64832ba10d91c18a30c41 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d02d3aef7a64832ba10d91c18a30c41; no voters:
I20260925 15:17:49.436137 31622 leader_election.cc:288] T 00000000000000000000000000000000 P 8d02d3aef7a64832ba10d91c18a30c41 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:17:49.436224 31627 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d02d3aef7a64832ba10d91c18a30c41 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:17:49.436954 31627 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d02d3aef7a64832ba10d91c18a30c41 [term 1 LEADER]: Becoming Leader. State: Replica: 8d02d3aef7a64832ba10d91c18a30c41, State: Running, Role: LEADER
I20260925 15:17:49.437525 31627 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d02d3aef7a64832ba10d91c18a30c41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d02d3aef7a64832ba10d91c18a30c41" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 45791 } }
I20260925 15:17:49.438232 31622 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d02d3aef7a64832ba10d91c18a30c41 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:17:49.440037 31628 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d02d3aef7a64832ba10d91c18a30c41 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d02d3aef7a64832ba10d91c18a30c41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d02d3aef7a64832ba10d91c18a30c41" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 45791 } } }
I20260925 15:17:49.440212 31629 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d02d3aef7a64832ba10d91c18a30c41 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d02d3aef7a64832ba10d91c18a30c41. Latest consensus state: current_term: 1 leader_uuid: "8d02d3aef7a64832ba10d91c18a30c41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d02d3aef7a64832ba10d91c18a30c41" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 45791 } } }
I20260925 15:17:49.440440 31628 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d02d3aef7a64832ba10d91c18a30c41 [sys.catalog]: This master's current role is: LEADER
I20260925 15:17:49.440503 31629 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d02d3aef7a64832ba10d91c18a30c41 [sys.catalog]: This master's current role is: LEADER
I20260925 15:17:49.442471 31632 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:17:49.447418 31632 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:17:49.456067 31632 catalog_manager.cc:1347] Generated new cluster ID: 75b9752c24174ff98c071d67696e8718
I20260925 15:17:49.456188 31632 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:17:49.481757 31632 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:17:49.483157 31632 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:17:49.489217 31632 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d02d3aef7a64832ba10d91c18a30c41: Generated new TSK 0
I20260925 15:17:49.490326 31632 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.8.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.1.keytab.
Entry for principal kudu/127.12.8.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.1.keytab.
I20260925 15:17:49.558465 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349162370577-12320-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349162370577-12320-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349162370577-12320-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349162370577-12320-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.1:0
--local_ip_for_outbound_sockets=127.12.8.1
--webserver_interface=127.12.8.1
--webserver_port=0
--tserver_master_addrs=127.12.8.62:45791
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.1.keytab
--principal=kudu/127.12.8.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:34103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:17:49.886732 31648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:17:49.887022 31648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:17:49.887123 31648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:17:49.894515 31648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:17:49.894650 31648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:17:49.894768 31648 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.1
I20260925 15:17:49.904560 31648 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:34103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349162370577-12320-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349162370577-12320-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.1.keytab
--principal=kudu/127.12.8.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349162370577-12320-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.8.1
--webserver_port=0
--tserver_master_addrs=127.12.8.62:45791
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.1
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349162370577-12320-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:17:49.905820 31648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:17:49.907168 31648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:17:49.921396 31654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:17:49.921481 31656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:17:49.921425 31653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:17:49.922297 31648 server_base.cc:1034] running on GCE node
I20260925 15:17:49.922904 31648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:17:49.923918 31648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:17:49.925168 31648 hybrid_clock.cc:648] HybridClock initialized: now 1790349469925105 us; error 60 us; skew 500 ppm
I20260925 15:17:49.925513 31648 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:17:49 dist-test-slave-p468 krb5kdc[31542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349469, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:17:49.928754 31648 init.cc:377] Logged in from keytab as kudu/127.12.8.1@KRBTEST.COM (short username kudu)
I20260925 15:17:49.930387 31648 webserver.cc:460] Webserver started at http://127.12.8.1:41309/ using document root <none> and password file <none>
I20260925 15:17:49.931039 31648 fs_manager.cc:359] Metadata directory not provided
I20260925 15:17:49.931138 31648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:17:49.931396 31648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:17:49.934260 31648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349162370577-12320-0/minicluster-data/ts-0/data/instance:
uuid: "96b4160a9ac4407a894078edfb23e1c6"
format_stamp: "Formatted at 2026-09-25 15:17:49 on dist-test-slave-p468"
server_key: "68dc60131e40c2a1ee2d636d8b77aeb5"
server_key_iv: "d51b7164a26f2b7975caf52e00a65635"
server_key_version: "encryptionkey@0"
I20260925 15:17:49.934991 31648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349162370577-12320-0/minicluster-data/ts-0/wal/instance:
uuid: "96b4160a9ac4407a894078edfb23e1c6"
format_stamp: "Formatted at 2026-09-25 15:17:49 on dist-test-slave-p468"
server_key: "68dc60131e40c2a1ee2d636d8b77aeb5"
server_key_iv: "d51b7164a26f2b7975caf52e00a65635"
server_key_version: "encryptionkey@0"
I20260925 15:17:49.940214 31648 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 15:17:49.943580 31663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:17:49.945043 31648 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 15:17:49.945216 31648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349162370577-12320-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349162370577-12320-0/minicluster-data/ts-0/wal
uuid: "96b4160a9ac4407a894078edfb23e1c6"
format_stamp: "Formatted at 2026-09-25 15:17:49 on dist-test-slave-p468"
server_key: "68dc60131e40c2a1ee2d636d8b77aeb5"
server_key_iv: "d51b7164a26f2b7975caf52e00a65635"
server_key_version: "encryptionkey@0"
I20260925 15:17:49.945418 31648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349162370577-12320-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349162370577-12320-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349162370577-12320-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:17:49.991078 31648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:17:49.995782 31648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:17:49.996101 31648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:17:49.997129 31648 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:17:49.998553 31648 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:17:49.998634 31648 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:17:49.998799 31648 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:17:49.998858 31648 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:17:50.033100 31648 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.1:34941
I20260925 15:17:50.033149 31775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.1:34941 every 8 connection(s)
I20260925 15:17:50.034426 31648 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349162370577-12320-0/minicluster-data/ts-0/data/info.pb
I20260925 15:17:50.043114 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 31648
I20260925 15:17:50.043325 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349162370577-12320-0/minicluster-data/ts-0/wal/instance
I20260925 15:17:50.043861 12320 external_mini_cluster.cc:1428] Setting key 42f64a39346ae88bc4074947a15d849f
Sep 25 15:17:50 dist-test-slave-p468 krb5kdc[31542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349469, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.1@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
I20260925 15:17:50.056769 31778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:50.037005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.1:55347 (local address 127.12.8.62:45791)
0925 15:17:50.037499 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:50.037506 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:50.038595 (+ 1089us) server_negotiation.cc:407] Connection header received
0925 15:17:50.040386 (+ 1791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:50.040390 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:50.040462 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:50.040566 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:17:50.042139 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:50.042937 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:50.045048 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:50.045624 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:50.045794 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:50.050158 (+ 4364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:17:50.050165 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:17:50.050170 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:17:50.050272 (+ 102us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:17:50.053147 (+ 2875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:17:50.053944 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:17:50.053951 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:17:50.053956 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:17:50.054083 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:17:50.054646 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:17:50.054652 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:17:50.054657 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:17:50.054981 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:17:50.055114 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:50.056002 (+ 888us) server_negotiation.cc:299] Negotiation successful
0925 15:17:50.056291 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":250,"threads_started":1}
I20260925 15:17:50.057461 31779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:50.037569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:45791 (local address 127.12.8.1:55347)
0925 15:17:50.038428 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:50.038473 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 15:17:50.040151 (+ 1678us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:17:50.040820 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:17:50.040828 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:17:50.041371 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:17:50.041952 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:50.041969 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:50.043116 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:50.043121 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:50.044901 (+ 1780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:50.044913 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:50.045874 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:50.045880 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:50.046167 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:50.046907 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:17:50.046944 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:17:50.049918 (+ 2974us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:17:50.053284 (+ 3366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:17:50.053293 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:17:50.053310 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:17:50.053817 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:17:50.054224 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:17:50.054232 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:17:50.054238 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:17:50.054496 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:17:50.055123 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:17:50.055132 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:17:50.055601 (+ 469us) client_negotiation.cc:769] Sending connection context
0925 15:17:50.055833 (+ 232us) client_negotiation.cc:240] Negotiation successful
0925 15:17:50.056319 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":375,"threads_started":1}
I20260925 15:17:50.059141 31776 heartbeater.cc:347] Connected to a master server at 127.12.8.62:45791
I20260925 15:17:50.059475 31776 heartbeater.cc:464] Registering TS with master...
I20260925 15:17:50.060359 31776 heartbeater.cc:511] Master 127.12.8.62:45791 requested a full tablet report, sending...
I20260925 15:17:50.062654 31587 ts_manager.cc:194] Registered new tserver with Master: 96b4160a9ac4407a894078edfb23e1c6 (127.12.8.1:34941)
I20260925 15:17:50.064117 31587 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.1@KRBTEST.COM'} at 127.12.8.1:55347
WARNING: no policy specified for kudu/127.12.8.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.2.keytab.
Entry for principal kudu/127.12.8.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.2.keytab.
I20260925 15:17:50.103801 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349162370577-12320-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349162370577-12320-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349162370577-12320-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349162370577-12320-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.2:0
--local_ip_for_outbound_sockets=127.12.8.2
--webserver_interface=127.12.8.2
--webserver_port=0
--tserver_master_addrs=127.12.8.62:45791
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.2.keytab
--principal=kudu/127.12.8.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:34103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:17:50.429307 31782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:17:50.429602 31782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:17:50.429785 31782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:17:50.437891 31782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:17:50.438002 31782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:17:50.438243 31782 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.2
I20260925 15:17:50.449281 31782 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:34103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349162370577-12320-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349162370577-12320-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.2.keytab
--principal=kudu/127.12.8.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349162370577-12320-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.8.2
--webserver_port=0
--tserver_master_addrs=127.12.8.62:45791
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.2
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349162370577-12320-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:17:50.450654 31782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:17:50.452080 31782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:17:50.465696 31788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:17:50.465734 31790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:17:50.465775 31787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:17:50.466920 31782 server_base.cc:1034] running on GCE node
I20260925 15:17:50.467519 31782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:17:50.468470 31782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:17:50.469702 31782 hybrid_clock.cc:648] HybridClock initialized: now 1790349470469644 us; error 47 us; skew 500 ppm
I20260925 15:17:50.469971 31782 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:17:50 dist-test-slave-p468 krb5kdc[31542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349470, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:17:50.473028 31782 init.cc:377] Logged in from keytab as kudu/127.12.8.2@KRBTEST.COM (short username kudu)
I20260925 15:17:50.474417 31782 webserver.cc:460] Webserver started at http://127.12.8.2:34859/ using document root <none> and password file <none>
I20260925 15:17:50.475023 31782 fs_manager.cc:359] Metadata directory not provided
I20260925 15:17:50.475116 31782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:17:50.475354 31782 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:17:50.478034 31782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349162370577-12320-0/minicluster-data/ts-1/data/instance:
uuid: "f2261dd3d0664afca6e831ead054741d"
format_stamp: "Formatted at 2026-09-25 15:17:50 on dist-test-slave-p468"
server_key: "02f37ff7ea28c29cfc65ffe8fe7860ac"
server_key_iv: "d85e2baf847d3374288b87e03e31d386"
server_key_version: "encryptionkey@0"
I20260925 15:17:50.478794 31782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349162370577-12320-0/minicluster-data/ts-1/wal/instance:
uuid: "f2261dd3d0664afca6e831ead054741d"
format_stamp: "Formatted at 2026-09-25 15:17:50 on dist-test-slave-p468"
server_key: "02f37ff7ea28c29cfc65ffe8fe7860ac"
server_key_iv: "d85e2baf847d3374288b87e03e31d386"
server_key_version: "encryptionkey@0"
I20260925 15:17:50.483518 31782 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260925 15:17:50.487098 31797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:17:50.488413 31782 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 15:17:50.488539 31782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349162370577-12320-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349162370577-12320-0/minicluster-data/ts-1/wal
uuid: "f2261dd3d0664afca6e831ead054741d"
format_stamp: "Formatted at 2026-09-25 15:17:50 on dist-test-slave-p468"
server_key: "02f37ff7ea28c29cfc65ffe8fe7860ac"
server_key_iv: "d85e2baf847d3374288b87e03e31d386"
server_key_version: "encryptionkey@0"
I20260925 15:17:50.488724 31782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349162370577-12320-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349162370577-12320-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349162370577-12320-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:17:50.519459 31782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:17:50.523849 31782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:17:50.524117 31782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:17:50.525074 31782 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:17:50.526585 31782 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:17:50.526669 31782 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:17:50.526759 31782 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:17:50.526803 31782 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:17:50.561429 31782 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.2:45389
I20260925 15:17:50.561477 31909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.2:45389 every 8 connection(s)
I20260925 15:17:50.562827 31782 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349162370577-12320-0/minicluster-data/ts-1/data/info.pb
I20260925 15:17:50.567523 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 31782
I20260925 15:17:50.567710 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349162370577-12320-0/minicluster-data/ts-1/wal/instance
I20260925 15:17:50.568238 12320 external_mini_cluster.cc:1428] Setting key 28d955ddc002e8b6d64fd5c2d4524a86
Sep 25 15:17:50 dist-test-slave-p468 krb5kdc[31542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349470, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.2@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
I20260925 15:17:50.584144 31912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:50.565362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.2:56181 (local address 127.12.8.62:45791)
0925 15:17:50.565766 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:50.565771 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:50.566818 (+ 1047us) server_negotiation.cc:407] Connection header received
0925 15:17:50.569134 (+ 2316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:50.569140 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:50.569228 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:50.569339 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:17:50.571055 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:50.572256 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:50.573982 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:50.574734 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:50.574862 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:50.578274 (+ 3412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:17:50.578282 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:17:50.578290 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:17:50.578339 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:17:50.580910 (+ 2571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:17:50.581856 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:17:50.581860 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:17:50.581863 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:17:50.581989 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:17:50.582497 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:17:50.582500 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:17:50.582502 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:17:50.582761 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:17:50.582873 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:50.583719 (+ 846us) server_negotiation.cc:299] Negotiation successful
0925 15:17:50.583880 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":202,"threads_started":1}
I20260925 15:17:50.585634 31913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:50.565959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:45791 (local address 127.12.8.2:56181)
0925 15:17:50.566661 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:50.566697 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 15:17:50.568891 (+ 2194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:17:50.569625 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:17:50.569636 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:17:50.570343 (+ 707us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:17:50.570883 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:50.570900 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:50.572438 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:50.572442 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:50.573829 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:50.573842 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:50.574952 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:50.574957 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:50.575162 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:50.575917 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:17:50.575938 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:17:50.578022 (+ 2084us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:17:50.581051 (+ 3029us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:17:50.581062 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:17:50.581081 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:17:50.581726 (+ 645us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:17:50.582108 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:17:50.582114 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:17:50.582121 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:17:50.582383 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:17:50.582887 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:17:50.582897 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:17:50.583323 (+ 426us) client_negotiation.cc:769] Sending connection context
0925 15:17:50.583639 (+ 316us) client_negotiation.cc:240] Negotiation successful
0925 15:17:50.584186 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":310,"threads_started":1}
I20260925 15:17:50.587235 31910 heartbeater.cc:347] Connected to a master server at 127.12.8.62:45791
I20260925 15:17:50.587568 31910 heartbeater.cc:464] Registering TS with master...
I20260925 15:17:50.588338 31910 heartbeater.cc:511] Master 127.12.8.62:45791 requested a full tablet report, sending...
I20260925 15:17:50.589819 31587 ts_manager.cc:194] Registered new tserver with Master: f2261dd3d0664afca6e831ead054741d (127.12.8.2:45389)
I20260925 15:17:50.590758 31587 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.2@KRBTEST.COM'} at 127.12.8.2:56181
WARNING: no policy specified for kudu/127.12.8.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.3.keytab.
Entry for principal kudu/127.12.8.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.3.keytab.
I20260925 15:17:50.625559 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349162370577-12320-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349162370577-12320-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349162370577-12320-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349162370577-12320-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.3:0
--local_ip_for_outbound_sockets=127.12.8.3
--webserver_interface=127.12.8.3
--webserver_port=0
--tserver_master_addrs=127.12.8.62:45791
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.3.keytab
--principal=kudu/127.12.8.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:34103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:17:50.957226 31916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:17:50.957597 31916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:17:50.957782 31916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:17:50.966109 31916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:17:50.966302 31916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:17:50.966486 31916 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.3
I20260925 15:17:50.976172 31916 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:34103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349162370577-12320-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349162370577-12320-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.3.keytab
--principal=kudu/127.12.8.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349162370577-12320-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.8.3
--webserver_port=0
--tserver_master_addrs=127.12.8.62:45791
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.3
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349162370577-12320-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:17:50.977445 31916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:17:50.978859 31916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:17:50.992522 31921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:17:50.992506 31922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:17:50.993221 31924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:17:50.993759 31916 server_base.cc:1034] running on GCE node
I20260925 15:17:50.994484 31916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:17:50.995477 31916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:17:50.996719 31916 hybrid_clock.cc:648] HybridClock initialized: now 1790349470996673 us; error 43 us; skew 500 ppm
I20260925 15:17:50.997051 31916 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:17:50 dist-test-slave-p468 krb5kdc[31542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349470, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:17:51.000407 31916 init.cc:377] Logged in from keytab as kudu/127.12.8.3@KRBTEST.COM (short username kudu)
I20260925 15:17:51.001894 31916 webserver.cc:460] Webserver started at http://127.12.8.3:34245/ using document root <none> and password file <none>
I20260925 15:17:51.002681 31916 fs_manager.cc:359] Metadata directory not provided
I20260925 15:17:51.002830 31916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:17:51.003144 31916 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:17:51.006139 31916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349162370577-12320-0/minicluster-data/ts-2/data/instance:
uuid: "d65251451b384c6dbc3643e1f4d6a10b"
format_stamp: "Formatted at 2026-09-25 15:17:51 on dist-test-slave-p468"
server_key: "b8b0fd50f0c91e83ea77b62a0fe612dd"
server_key_iv: "b80621838cb63bef3a7f6115c2e51935"
server_key_version: "encryptionkey@0"
I20260925 15:17:51.006995 31916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349162370577-12320-0/minicluster-data/ts-2/wal/instance:
uuid: "d65251451b384c6dbc3643e1f4d6a10b"
format_stamp: "Formatted at 2026-09-25 15:17:51 on dist-test-slave-p468"
server_key: "b8b0fd50f0c91e83ea77b62a0fe612dd"
server_key_iv: "b80621838cb63bef3a7f6115c2e51935"
server_key_version: "encryptionkey@0"
I20260925 15:17:51.012228 31916 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 15:17:51.015859 31931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:17:51.017213 31916 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 15:17:51.017398 31916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349162370577-12320-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349162370577-12320-0/minicluster-data/ts-2/wal
uuid: "d65251451b384c6dbc3643e1f4d6a10b"
format_stamp: "Formatted at 2026-09-25 15:17:51 on dist-test-slave-p468"
server_key: "b8b0fd50f0c91e83ea77b62a0fe612dd"
server_key_iv: "b80621838cb63bef3a7f6115c2e51935"
server_key_version: "encryptionkey@0"
I20260925 15:17:51.017618 31916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349162370577-12320-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349162370577-12320-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349162370577-12320-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:17:51.043315 31916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:17:51.047509 31916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:17:51.047761 31916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:17:51.048767 31916 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:17:51.050180 31916 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:17:51.050312 31916 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260925 15:17:51.050398 31916 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:17:51.050448 31916 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:17:51.067970 31776 heartbeater.cc:503] Master 127.12.8.62:45791 was elected leader, sending a full tablet report...
I20260925 15:17:51.085223 31916 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.3:39713
I20260925 15:17:51.085286 32043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.3:39713 every 8 connection(s)
I20260925 15:17:51.086632 31916 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349162370577-12320-0/minicluster-data/ts-2/data/info.pb
I20260925 15:17:51.089128 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 31916
I20260925 15:17:51.089324 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349162370577-12320-0/minicluster-data/ts-2/wal/instance
I20260925 15:17:51.089922 12320 external_mini_cluster.cc:1428] Setting key 929ad77adae334a9c05d9c0025cc38f7
Sep 25 15:17:51 dist-test-slave-p468 krb5kdc[31542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349470, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.3@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
I20260925 15:17:51.107389 32046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:51.089289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.3:58557 (local address 127.12.8.62:45791)
0925 15:17:51.089766 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:51.089773 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:51.090852 (+ 1079us) server_negotiation.cc:407] Connection header received
0925 15:17:51.092871 (+ 2019us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:51.092876 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:51.092984 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:51.093104 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:17:51.094749 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:51.095499 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:51.097056 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:51.097473 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:51.097614 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:51.101037 (+ 3423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:17:51.101041 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:17:51.101045 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:17:51.101078 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:17:51.104375 (+ 3297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:17:51.105251 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:17:51.105256 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:17:51.105259 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:17:51.105416 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:17:51.105800 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:17:51.105803 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:17:51.105806 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:17:51.106076 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:17:51.106258 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:51.106951 (+ 693us) server_negotiation.cc:299] Negotiation successful
0925 15:17:51.107128 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":204,"threads_started":1}
I20260925 15:17:51.108341 32047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:51.089829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:45791 (local address 127.12.8.3:58557)
0925 15:17:51.090694 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:51.090747 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 15:17:51.092604 (+ 1857us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:17:51.093360 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:17:51.093368 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:17:51.093942 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:17:51.094579 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:51.094596 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:51.095659 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:51.095664 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:51.096893 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:51.096905 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:51.097711 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:51.097718 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:51.097947 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:51.098629 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:17:51.098651 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:17:51.100842 (+ 2191us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:17:51.104517 (+ 3675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:17:51.104525 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:17:51.104541 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:17:51.105099 (+ 558us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:17:51.105504 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:17:51.105507 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:17:51.105510 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:17:51.105698 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:17:51.106240 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:17:51.106248 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:17:51.106565 (+ 317us) client_negotiation.cc:769] Sending connection context
0925 15:17:51.106830 (+ 265us) client_negotiation.cc:240] Negotiation successful
0925 15:17:51.107199 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":406,"threads_started":1}
I20260925 15:17:51.109771 32044 heartbeater.cc:347] Connected to a master server at 127.12.8.62:45791
I20260925 15:17:51.110097 32044 heartbeater.cc:464] Registering TS with master...
I20260925 15:17:51.110869 32044 heartbeater.cc:511] Master 127.12.8.62:45791 requested a full tablet report, sending...
I20260925 15:17:51.112360 31587 ts_manager.cc:194] Registered new tserver with Master: d65251451b384c6dbc3643e1f4d6a10b (127.12.8.3:39713)
I20260925 15:17:51.113183 31587 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.8.3@KRBTEST.COM'} at 127.12.8.3:58557
I20260925 15:17:51.117758 12320 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:17:51.134248 32046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:51.121912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53458 (local address 127.12.8.62:45791)
0925 15:17:51.122077 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:51.122081 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:51.122655 (+ 574us) server_negotiation.cc:407] Connection header received
0925 15:17:51.122789 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:51.122792 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:51.122853 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:51.122935 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:17:51.124315 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:51.125060 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:51.126385 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:51.126762 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:51.126855 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:51.129104 (+ 2249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:17:51.129109 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:17:51.129112 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:17:51.129139 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:17:51.131578 (+ 2439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:17:51.132268 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:17:51.132273 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:17:51.132282 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:17:51.132436 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:17:51.132876 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:17:51.132879 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:17:51.132881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:17:51.133182 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:17:51.133598 (+ 416us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:51.133805 (+ 207us) server_negotiation.cc:299] Negotiation successful
0925 15:17:51.133971 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 15:17:51.137905 31587 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53458:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:17:51.145331 31587 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:17:51.179734 32058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:51.158651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50130 (local address 127.12.8.1:34941)
0925 15:17:51.171100 (+ 12449us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:51.171115 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:51.171152 (+ 37us) server_negotiation.cc:407] Connection header received
0925 15:17:51.171264 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:51.171272 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:51.171587 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:51.171823 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:17:51.172454 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:51.174713 (+ 2259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:51.176987 (+ 2274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:51.178422 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:51.178614 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:51.178833 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:51.179164 (+ 331us) server_negotiation.cc:299] Negotiation successful
0925 15:17:51.179339 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12197,"thread_start_us":11989,"threads_started":1}
I20260925 15:17:51.179811 32060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:51.158982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.1:34941 (local address 127.0.0.1:50130)
0925 15:17:51.159521 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:51.159535 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 15:17:51.159631 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:17:51.171832 (+ 12201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:17:51.171839 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:17:51.171869 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:17:51.172254 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:51.172267 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:51.174982 (+ 2715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:51.174987 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:51.176743 (+ 1756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:51.176756 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:51.178685 (+ 1929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:51.178691 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:51.178880 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:51.178934 (+ 54us) client_negotiation.cc:769] Sending connection context
0925 15:17:51.179094 (+ 160us) client_negotiation.cc:240] Negotiation successful
0925 15:17:51.179352 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"mutex_wait_us":33,"thread_start_us":322,"threads_started":1}
I20260925 15:17:51.180200 32062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:51.160255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55100 (local address 127.12.8.2:45389)
0925 15:17:51.172175 (+ 11920us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:51.172183 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:51.172216 (+ 33us) server_negotiation.cc:407] Connection header received
0925 15:17:51.172289 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:51.172293 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:51.172462 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:51.172660 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:17:51.173253 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:51.174292 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:51.176885 (+ 2593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:51.178426 (+ 1541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:51.178582 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:51.178798 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:51.179739 (+ 941us) server_negotiation.cc:299] Negotiation successful
0925 15:17:51.179938 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11771,"thread_start_us":11660,"threads_started":1}
I20260925 15:17:51.180210 32057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:51.157973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.3:39713 (local address 127.0.0.1:34442)
0925 15:17:51.158849 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:51.158891 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 15:17:51.159094 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:17:51.173097 (+ 14003us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:17:51.173102 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:17:51.173131 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:17:51.173482 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:51.173492 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:51.175677 (+ 2185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:51.175683 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:51.177479 (+ 1796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:51.177492 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:51.179326 (+ 1834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:51.179332 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:51.179508 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:51.179545 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 15:17:51.179609 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 15:17:51.179735 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"thread_start_us":522,"threads_started":1}
I20260925 15:17:51.180645 32061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:51.159973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.2:45389 (local address 127.0.0.1:55100)
0925 15:17:51.171477 (+ 11504us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:51.171505 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0925 15:17:51.171670 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:17:51.172643 (+ 973us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:17:51.172647 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:17:51.172672 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:17:51.173055 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:51.173069 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:51.174449 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:51.174453 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:51.176700 (+ 2247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:51.176718 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:51.178950 (+ 2232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:51.178955 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:51.179118 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:51.179149 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 15:17:51.179236 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 15:17:51.179384 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11334,"mutex_wait_us":32,"thread_start_us":11145,"threads_started":1}
I20260925 15:17:51.181028 32059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:51.158730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34442 (local address 127.12.8.3:39713)
0925 15:17:51.172491 (+ 13761us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:51.172499 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:51.172532 (+ 33us) server_negotiation.cc:407] Connection header received
0925 15:17:51.172643 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:51.172650 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:51.172882 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:51.173671 (+ 789us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:17:51.173953 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:51.175517 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:51.177662 (+ 2145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:51.179097 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:51.179942 (+ 845us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:51.180168 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:51.180242 (+ 74us) server_negotiation.cc:299] Negotiation successful
0925 15:17:51.180710 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13587,"thread_start_us":14149,"threads_started":1}
I20260925 15:17:51.184381 31979 tablet_service.cc:1467] Processing CreateTablet for tablet ace54e4655b0450fa319e57351cc5268 (DEFAULT_TABLE table=test-table [id=b1b387ed9810458fbab2bd9d08a83179]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:17:51.184458 31711 tablet_service.cc:1467] Processing CreateTablet for tablet ace54e4655b0450fa319e57351cc5268 (DEFAULT_TABLE table=test-table [id=b1b387ed9810458fbab2bd9d08a83179]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:17:51.184644 31845 tablet_service.cc:1467] Processing CreateTablet for tablet ace54e4655b0450fa319e57351cc5268 (DEFAULT_TABLE table=test-table [id=b1b387ed9810458fbab2bd9d08a83179]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:17:51.185906 31845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ace54e4655b0450fa319e57351cc5268. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:17:51.185906 31711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ace54e4655b0450fa319e57351cc5268. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:17:51.185921 31979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ace54e4655b0450fa319e57351cc5268. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:17:51.197062 32063 tablet_bootstrap.cc:492] T ace54e4655b0450fa319e57351cc5268 P d65251451b384c6dbc3643e1f4d6a10b: Bootstrap starting.
I20260925 15:17:51.200373 32063 tablet_bootstrap.cc:654] T ace54e4655b0450fa319e57351cc5268 P d65251451b384c6dbc3643e1f4d6a10b: Neither blocks nor log segments found. Creating new log.
I20260925 15:17:51.201300 32064 tablet_bootstrap.cc:492] T ace54e4655b0450fa319e57351cc5268 P 96b4160a9ac4407a894078edfb23e1c6: Bootstrap starting.
I20260925 15:17:51.201848 32063 log.cc:824] T ace54e4655b0450fa319e57351cc5268 P d65251451b384c6dbc3643e1f4d6a10b: Log is configured to *not* fsync() on all Append() calls
I20260925 15:17:51.203297 32065 tablet_bootstrap.cc:492] T ace54e4655b0450fa319e57351cc5268 P f2261dd3d0664afca6e831ead054741d: Bootstrap starting.
I20260925 15:17:51.204972 32064 tablet_bootstrap.cc:654] T ace54e4655b0450fa319e57351cc5268 P 96b4160a9ac4407a894078edfb23e1c6: Neither blocks nor log segments found. Creating new log.
I20260925 15:17:51.206205 32064 log.cc:824] T ace54e4655b0450fa319e57351cc5268 P 96b4160a9ac4407a894078edfb23e1c6: Log is configured to *not* fsync() on all Append() calls
I20260925 15:17:51.206709 32065 tablet_bootstrap.cc:654] T ace54e4655b0450fa319e57351cc5268 P f2261dd3d0664afca6e831ead054741d: Neither blocks nor log segments found. Creating new log.
I20260925 15:17:51.207811 32065 log.cc:824] T ace54e4655b0450fa319e57351cc5268 P f2261dd3d0664afca6e831ead054741d: Log is configured to *not* fsync() on all Append() calls
I20260925 15:17:51.208047 32063 tablet_bootstrap.cc:492] T ace54e4655b0450fa319e57351cc5268 P d65251451b384c6dbc3643e1f4d6a10b: No bootstrap required, opened a new log
I20260925 15:17:51.208351 32063 ts_tablet_manager.cc:1397] T ace54e4655b0450fa319e57351cc5268 P d65251451b384c6dbc3643e1f4d6a10b: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260925 15:17:51.209597 32064 tablet_bootstrap.cc:492] T ace54e4655b0450fa319e57351cc5268 P 96b4160a9ac4407a894078edfb23e1c6: No bootstrap required, opened a new log
I20260925 15:17:51.209852 32064 ts_tablet_manager.cc:1397] T ace54e4655b0450fa319e57351cc5268 P 96b4160a9ac4407a894078edfb23e1c6: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260925 15:17:51.210935 32065 tablet_bootstrap.cc:492] T ace54e4655b0450fa319e57351cc5268 P f2261dd3d0664afca6e831ead054741d: No bootstrap required, opened a new log
I20260925 15:17:51.211182 32065 ts_tablet_manager.cc:1397] T ace54e4655b0450fa319e57351cc5268 P f2261dd3d0664afca6e831ead054741d: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260925 15:17:51.215734 32063 raft_consensus.cc:348] T ace54e4655b0450fa319e57351cc5268 P d65251451b384c6dbc3643e1f4d6a10b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d65251451b384c6dbc3643e1f4d6a10b" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 39713 } } peers { permanent_uuid: "96b4160a9ac4407a894078edfb23e1c6" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 34941 } } peers { permanent_uuid: "f2261dd3d0664afca6e831ead054741d" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 45389 } }
I20260925 15:17:51.216102 32063 raft_consensus.cc:374] T ace54e4655b0450fa319e57351cc5268 P d65251451b384c6dbc3643e1f4d6a10b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:17:51.216194 32063 raft_consensus.cc:729] T ace54e4655b0450fa319e57351cc5268 P d65251451b384c6dbc3643e1f4d6a10b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d65251451b384c6dbc3643e1f4d6a10b, State: Initialized, Role: FOLLOWER
I20260925 15:17:51.216826 32063 consensus_queue.cc:260] T ace54e4655b0450fa319e57351cc5268 P d65251451b384c6dbc3643e1f4d6a10b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d65251451b384c6dbc3643e1f4d6a10b" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 39713 } } peers { permanent_uuid: "96b4160a9ac4407a894078edfb23e1c6" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 34941 } } peers { permanent_uuid: "f2261dd3d0664afca6e831ead054741d" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 45389 } }
I20260925 15:17:51.217893 32044 heartbeater.cc:503] Master 127.12.8.62:45791 was elected leader, sending a full tablet report...
I20260925 15:17:51.218679 32063 ts_tablet_manager.cc:1428] T ace54e4655b0450fa319e57351cc5268 P d65251451b384c6dbc3643e1f4d6a10b: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260925 15:17:51.220456 32064 raft_consensus.cc:348] T ace54e4655b0450fa319e57351cc5268 P 96b4160a9ac4407a894078edfb23e1c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d65251451b384c6dbc3643e1f4d6a10b" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 39713 } } peers { permanent_uuid: "96b4160a9ac4407a894078edfb23e1c6" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 34941 } } peers { permanent_uuid: "f2261dd3d0664afca6e831ead054741d" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 45389 } }
I20260925 15:17:51.220875 32064 raft_consensus.cc:374] T ace54e4655b0450fa319e57351cc5268 P 96b4160a9ac4407a894078edfb23e1c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:17:51.220983 32064 raft_consensus.cc:729] T ace54e4655b0450fa319e57351cc5268 P 96b4160a9ac4407a894078edfb23e1c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96b4160a9ac4407a894078edfb23e1c6, State: Initialized, Role: FOLLOWER
I20260925 15:17:51.221608 32064 consensus_queue.cc:260] T ace54e4655b0450fa319e57351cc5268 P 96b4160a9ac4407a894078edfb23e1c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d65251451b384c6dbc3643e1f4d6a10b" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 39713 } } peers { permanent_uuid: "96b4160a9ac4407a894078edfb23e1c6" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 34941 } } peers { permanent_uuid: "f2261dd3d0664afca6e831ead054741d" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 45389 } }
I20260925 15:17:51.222677 32065 raft_consensus.cc:348] T ace54e4655b0450fa319e57351cc5268 P f2261dd3d0664afca6e831ead054741d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d65251451b384c6dbc3643e1f4d6a10b" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 39713 } } peers { permanent_uuid: "96b4160a9ac4407a894078edfb23e1c6" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 34941 } } peers { permanent_uuid: "f2261dd3d0664afca6e831ead054741d" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 45389 } }
I20260925 15:17:51.223088 32065 raft_consensus.cc:374] T ace54e4655b0450fa319e57351cc5268 P f2261dd3d0664afca6e831ead054741d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:17:51.223203 32065 raft_consensus.cc:729] T ace54e4655b0450fa319e57351cc5268 P f2261dd3d0664afca6e831ead054741d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2261dd3d0664afca6e831ead054741d, State: Initialized, Role: FOLLOWER
I20260925 15:17:51.223392 32064 ts_tablet_manager.cc:1428] T ace54e4655b0450fa319e57351cc5268 P 96b4160a9ac4407a894078edfb23e1c6: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260925 15:17:51.223789 32065 consensus_queue.cc:260] T ace54e4655b0450fa319e57351cc5268 P f2261dd3d0664afca6e831ead054741d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d65251451b384c6dbc3643e1f4d6a10b" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 39713 } } peers { permanent_uuid: "96b4160a9ac4407a894078edfb23e1c6" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 34941 } } peers { permanent_uuid: "f2261dd3d0664afca6e831ead054741d" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 45389 } }
I20260925 15:17:51.225090 31910 heartbeater.cc:503] Master 127.12.8.62:45791 was elected leader, sending a full tablet report...
I20260925 15:17:51.225646 32065 ts_tablet_manager.cc:1428] T ace54e4655b0450fa319e57351cc5268 P f2261dd3d0664afca6e831ead054741d: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260925 15:17:51.262317 32069 raft_consensus.cc:482] T ace54e4655b0450fa319e57351cc5268 P d65251451b384c6dbc3643e1f4d6a10b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:17:51.262626 32069 raft_consensus.cc:504] T ace54e4655b0450fa319e57351cc5268 P d65251451b384c6dbc3643e1f4d6a10b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d65251451b384c6dbc3643e1f4d6a10b" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 39713 } } peers { permanent_uuid: "96b4160a9ac4407a894078edfb23e1c6" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 34941 } } peers { permanent_uuid: "f2261dd3d0664afca6e831ead054741d" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 45389 } }
I20260925 15:17:51.264343 32069 leader_election.cc:288] T ace54e4655b0450fa319e57351cc5268 P d65251451b384c6dbc3643e1f4d6a10b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96b4160a9ac4407a894078edfb23e1c6 (127.12.8.1:34941), f2261dd3d0664afca6e831ead054741d (127.12.8.2:45389)
I20260925 15:17:51.270946 32058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:51.264706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.3:42351 (local address 127.12.8.1:34941)
0925 15:17:51.265325 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:51.265330 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:51.265345 (+ 15us) server_negotiation.cc:407] Connection header received
0925 15:17:51.265447 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:51.265454 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:51.265541 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:51.265655 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:17:51.266133 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:51.267307 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:51.269271 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:51.270033 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:51.270214 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:51.270318 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:51.270570 (+ 252us) server_negotiation.cc:299] Negotiation successful
0925 15:17:51.270695 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489}
I20260925 15:17:51.270989 32047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:51.264700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.1:34941 (local address 127.12.8.3:42351)
0925 15:17:51.264916 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:51.264934 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 15:17:51.265067 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:17:51.265741 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:17:51.265745 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:17:51.265777 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:17:51.265959 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:51.265969 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:51.267451 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:51.267457 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:51.269098 (+ 1641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:51.269111 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:51.270214 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:51.270220 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:51.270409 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:51.270433 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 15:17:51.270544 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 15:17:51.270675 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73,"mutex_wait_us":21}
I20260925 15:17:51.271514 32062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:51.265430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.3:59473 (local address 127.12.8.2:45389)
0925 15:17:51.265641 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:51.265647 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:51.265972 (+ 325us) server_negotiation.cc:407] Connection header received
0925 15:17:51.266104 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:51.266114 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:51.266175 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:51.266298 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:17:51.266908 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:51.267783 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:51.269693 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:51.270534 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:51.270704 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:51.270806 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:51.271175 (+ 369us) server_negotiation.cc:299] Negotiation successful
0925 15:17:51.271296 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260925 15:17:51.272097 32072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:51.265093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.2:45389 (local address 127.12.8.3:59473)
0925 15:17:51.265867 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:51.265884 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 15:17:51.265981 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:17:51.266336 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:17:51.266340 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:17:51.266364 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:17:51.266737 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:51.266749 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:51.267951 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:51.267955 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:51.269516 (+ 1561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:51.269529 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:51.270784 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:51.270790 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:51.270943 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:51.270966 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 15:17:51.271703 (+ 737us) client_negotiation.cc:240] Negotiation successful
0925 15:17:51.271831 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":651,"mutex_wait_us":21,"thread_start_us":429,"threads_started":1}
I20260925 15:17:51.271965 31731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ace54e4655b0450fa319e57351cc5268" candidate_uuid: "d65251451b384c6dbc3643e1f4d6a10b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96b4160a9ac4407a894078edfb23e1c6" is_pre_election: true
I20260925 15:17:51.272531 31731 raft_consensus.cc:2454] T ace54e4655b0450fa319e57351cc5268 P 96b4160a9ac4407a894078edfb23e1c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d65251451b384c6dbc3643e1f4d6a10b in term 0.
I20260925 15:17:51.273244 31933 leader_election.cc:302] T ace54e4655b0450fa319e57351cc5268 P d65251451b384c6dbc3643e1f4d6a10b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96b4160a9ac4407a894078edfb23e1c6, d65251451b384c6dbc3643e1f4d6a10b; no voters:
I20260925 15:17:51.273177 31865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ace54e4655b0450fa319e57351cc5268" candidate_uuid: "d65251451b384c6dbc3643e1f4d6a10b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2261dd3d0664afca6e831ead054741d" is_pre_election: true
I20260925 15:17:51.273689 32069 raft_consensus.cc:2781] T ace54e4655b0450fa319e57351cc5268 P d65251451b384c6dbc3643e1f4d6a10b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:17:51.273686 31865 raft_consensus.cc:2454] T ace54e4655b0450fa319e57351cc5268 P f2261dd3d0664afca6e831ead054741d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d65251451b384c6dbc3643e1f4d6a10b in term 0.
I20260925 15:17:51.273850 32069 raft_consensus.cc:482] T ace54e4655b0450fa319e57351cc5268 P d65251451b384c6dbc3643e1f4d6a10b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:17:51.273924 32069 raft_consensus.cc:3037] T ace54e4655b0450fa319e57351cc5268 P d65251451b384c6dbc3643e1f4d6a10b [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:17:51.276548 32069 raft_consensus.cc:504] T ace54e4655b0450fa319e57351cc5268 P d65251451b384c6dbc3643e1f4d6a10b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d65251451b384c6dbc3643e1f4d6a10b" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 39713 } } peers { permanent_uuid: "96b4160a9ac4407a894078edfb23e1c6" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 34941 } } peers { permanent_uuid: "f2261dd3d0664afca6e831ead054741d" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 45389 } }
I20260925 15:17:51.277217 32069 leader_election.cc:288] T ace54e4655b0450fa319e57351cc5268 P d65251451b384c6dbc3643e1f4d6a10b [CANDIDATE]: Term 1 election: Requested vote from peers 96b4160a9ac4407a894078edfb23e1c6 (127.12.8.1:34941), f2261dd3d0664afca6e831ead054741d (127.12.8.2:45389)
I20260925 15:17:51.277661 31731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ace54e4655b0450fa319e57351cc5268" candidate_uuid: "d65251451b384c6dbc3643e1f4d6a10b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96b4160a9ac4407a894078edfb23e1c6"
I20260925 15:17:51.277760 31865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ace54e4655b0450fa319e57351cc5268" candidate_uuid: "d65251451b384c6dbc3643e1f4d6a10b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2261dd3d0664afca6e831ead054741d"
I20260925 15:17:51.277871 31731 raft_consensus.cc:3037] T ace54e4655b0450fa319e57351cc5268 P 96b4160a9ac4407a894078edfb23e1c6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:17:51.277997 31865 raft_consensus.cc:3037] T ace54e4655b0450fa319e57351cc5268 P f2261dd3d0664afca6e831ead054741d [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:17:51.280252 31865 raft_consensus.cc:2454] T ace54e4655b0450fa319e57351cc5268 P f2261dd3d0664afca6e831ead054741d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d65251451b384c6dbc3643e1f4d6a10b in term 1.
I20260925 15:17:51.280288 31731 raft_consensus.cc:2454] T ace54e4655b0450fa319e57351cc5268 P 96b4160a9ac4407a894078edfb23e1c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d65251451b384c6dbc3643e1f4d6a10b in term 1.
I20260925 15:17:51.280644 31933 leader_election.cc:302] T ace54e4655b0450fa319e57351cc5268 P d65251451b384c6dbc3643e1f4d6a10b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96b4160a9ac4407a894078edfb23e1c6, d65251451b384c6dbc3643e1f4d6a10b; no voters:
I20260925 15:17:51.280918 32069 raft_consensus.cc:2781] T ace54e4655b0450fa319e57351cc5268 P d65251451b384c6dbc3643e1f4d6a10b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:17:51.281383 32069 raft_consensus.cc:686] T ace54e4655b0450fa319e57351cc5268 P d65251451b384c6dbc3643e1f4d6a10b [term 1 LEADER]: Becoming Leader. State: Replica: d65251451b384c6dbc3643e1f4d6a10b, State: Running, Role: LEADER
I20260925 15:17:51.281857 32069 consensus_queue.cc:237] T ace54e4655b0450fa319e57351cc5268 P d65251451b384c6dbc3643e1f4d6a10b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d65251451b384c6dbc3643e1f4d6a10b" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 39713 } } peers { permanent_uuid: "96b4160a9ac4407a894078edfb23e1c6" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 34941 } } peers { permanent_uuid: "f2261dd3d0664afca6e831ead054741d" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 45389 } }
W20260925 15:17:51.286825 31777 tablet.cc:2369] T ace54e4655b0450fa319e57351cc5268 P 96b4160a9ac4407a894078edfb23e1c6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:17:51.286787 31586 catalog_manager.cc:5495] T ace54e4655b0450fa319e57351cc5268 P d65251451b384c6dbc3643e1f4d6a10b reported cstate change: term changed from 0 to 1, leader changed from <none> to d65251451b384c6dbc3643e1f4d6a10b (127.12.8.3). New cstate: current_term: 1 leader_uuid: "d65251451b384c6dbc3643e1f4d6a10b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d65251451b384c6dbc3643e1f4d6a10b" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 39713 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96b4160a9ac4407a894078edfb23e1c6" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 34941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f2261dd3d0664afca6e831ead054741d" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 45389 } health_report { overall_health: UNKNOWN } } }
I20260925 15:17:51.303807 32059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:51.298497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34448 (local address 127.12.8.3:39713)
0925 15:17:51.298725 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:51.298730 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:51.298766 (+ 36us) server_negotiation.cc:407] Connection header received
0925 15:17:51.298971 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:51.298975 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:51.299064 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:51.299192 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:17:51.299534 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:51.300307 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:51.301647 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:51.302097 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:51.302282 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:51.302728 (+ 446us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:17:51.303203 (+ 475us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:17:51.303318 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:51.303395 (+ 77us) server_negotiation.cc:299] Negotiation successful
0925 15:17:51.303483 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 15:17:51.311739 31731 raft_consensus.cc:1261] T ace54e4655b0450fa319e57351cc5268 P 96b4160a9ac4407a894078edfb23e1c6 [term 1 FOLLOWER]: Refusing update from remote peer d65251451b384c6dbc3643e1f4d6a10b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:17:51.311772 31865 raft_consensus.cc:1261] T ace54e4655b0450fa319e57351cc5268 P f2261dd3d0664afca6e831ead054741d [term 1 FOLLOWER]: Refusing update from remote peer d65251451b384c6dbc3643e1f4d6a10b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:17:51.312700 32073 consensus_queue.cc:1035] T ace54e4655b0450fa319e57351cc5268 P d65251451b384c6dbc3643e1f4d6a10b [LEADER]: Connected to new peer: Peer: permanent_uuid: "96b4160a9ac4407a894078edfb23e1c6" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 34941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:17:51.313058 32069 consensus_queue.cc:1035] T ace54e4655b0450fa319e57351cc5268 P d65251451b384c6dbc3643e1f4d6a10b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2261dd3d0664afca6e831ead054741d" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 45389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 15:17:51.314872 31911 tablet.cc:2369] T ace54e4655b0450fa319e57351cc5268 P f2261dd3d0664afca6e831ead054741d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:17:51.324196 32076 mvcc.cc:205] Tried to move back new op lower bound from 7333271434482540544 to 7333271434374668288. Current Snapshot: MvccSnapshot[applied={T|T < 7333271434482540544}]
I20260925 15:17:51.327392 32077 mvcc.cc:205] Tried to move back new op lower bound from 7333271434482540544 to 7333271434374668288. Current Snapshot: MvccSnapshot[applied={T|T < 7333271434482540544}]
I20260925 15:17:51.337952 31587 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53458:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:17:51.338382 31587 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53458:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:17:51.341972 31587 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8d02d3aef7a64832ba10d91c18a30c41: Sending DeleteTablet for 3 replicas of tablet ace54e4655b0450fa319e57351cc5268
I20260925 15:17:51.343169 31979 tablet_service.cc:1514] Processing DeleteTablet for tablet ace54e4655b0450fa319e57351cc5268 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:17:51 UTC) from {username='kudu', principal='kudu/127.12.8.62@KRBTEST.COM'} at 127.0.0.1:34442
I20260925 15:17:51.343701 31845 tablet_service.cc:1514] Processing DeleteTablet for tablet ace54e4655b0450fa319e57351cc5268 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:17:51 UTC) from {username='kudu', principal='kudu/127.12.8.62@KRBTEST.COM'} at 127.0.0.1:55100
I20260925 15:17:51.344435 31711 tablet_service.cc:1514] Processing DeleteTablet for tablet ace54e4655b0450fa319e57351cc5268 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:17:51 UTC) from {username='kudu', principal='kudu/127.12.8.62@KRBTEST.COM'} at 127.0.0.1:50130
I20260925 15:17:51.344588 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 31648
I20260925 15:17:51.355949 32085 tablet_replica.cc:330] T ace54e4655b0450fa319e57351cc5268 P d65251451b384c6dbc3643e1f4d6a10b: stopping tablet replica
I20260925 15:17:51.356577 32085 raft_consensus.cc:2229] T ace54e4655b0450fa319e57351cc5268 P d65251451b384c6dbc3643e1f4d6a10b [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:17:51.357412 32085 raft_consensus.cc:2258] T ace54e4655b0450fa319e57351cc5268 P d65251451b384c6dbc3643e1f4d6a10b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:17:51.359730 32085 ts_tablet_manager.cc:1905] T ace54e4655b0450fa319e57351cc5268 P d65251451b384c6dbc3643e1f4d6a10b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:17:51.362128 32086 tablet_replica.cc:330] T ace54e4655b0450fa319e57351cc5268 P f2261dd3d0664afca6e831ead054741d: stopping tablet replica
I20260925 15:17:51.362587 32086 raft_consensus.cc:2229] T ace54e4655b0450fa319e57351cc5268 P f2261dd3d0664afca6e831ead054741d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:17:51.363121 32086 raft_consensus.cc:2258] T ace54e4655b0450fa319e57351cc5268 P f2261dd3d0664afca6e831ead054741d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:17:51.363871 32087 tablet_replica.cc:330] T ace54e4655b0450fa319e57351cc5268 P 96b4160a9ac4407a894078edfb23e1c6: stopping tablet replica
I20260925 15:17:51.364300 32087 raft_consensus.cc:2229] T ace54e4655b0450fa319e57351cc5268 P 96b4160a9ac4407a894078edfb23e1c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:17:51.364835 32087 raft_consensus.cc:2258] T ace54e4655b0450fa319e57351cc5268 P 96b4160a9ac4407a894078edfb23e1c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:17:51.367187 32087 ts_tablet_manager.cc:1905] T ace54e4655b0450fa319e57351cc5268 P 96b4160a9ac4407a894078edfb23e1c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:17:51.367880 32085 ts_tablet_manager.cc:1918] T ace54e4655b0450fa319e57351cc5268 P d65251451b384c6dbc3643e1f4d6a10b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:17:51.368238 32085 log.cc:1196] T ace54e4655b0450fa319e57351cc5268 P d65251451b384c6dbc3643e1f4d6a10b: Deleting WAL directory at /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349162370577-12320-0/minicluster-data/ts-2/wal/wals/ace54e4655b0450fa319e57351cc5268
I20260925 15:17:51.368851 32085 ts_tablet_manager.cc:1939] T ace54e4655b0450fa319e57351cc5268 P d65251451b384c6dbc3643e1f4d6a10b: Deleting consensus metadata
Sep 25 15:17:51 dist-test-slave-p468 krb5kdc[31542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.1@KRBTEST.COM
I20260925 15:17:51.370857 31573 catalog_manager.cc:4841] TS d65251451b384c6dbc3643e1f4d6a10b (127.12.8.3:39713): tablet ace54e4655b0450fa319e57351cc5268 (table test-table [id=b1b387ed9810458fbab2bd9d08a83179]) successfully deleted
I20260925 15:17:51.371477 32086 ts_tablet_manager.cc:1905] T ace54e4655b0450fa319e57351cc5268 P f2261dd3d0664afca6e831ead054741d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:17:51.373181 32087 ts_tablet_manager.cc:1918] T ace54e4655b0450fa319e57351cc5268 P 96b4160a9ac4407a894078edfb23e1c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:17:51.373353 32087 log.cc:1196] T ace54e4655b0450fa319e57351cc5268 P 96b4160a9ac4407a894078edfb23e1c6: Deleting WAL directory at /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349162370577-12320-0/minicluster-data/ts-0/wal/wals/ace54e4655b0450fa319e57351cc5268
I20260925 15:17:51.373795 32087 ts_tablet_manager.cc:1939] T ace54e4655b0450fa319e57351cc5268 P 96b4160a9ac4407a894078edfb23e1c6: Deleting consensus metadata
I20260925 15:17:51.375664 31572 catalog_manager.cc:4841] TS 96b4160a9ac4407a894078edfb23e1c6 (127.12.8.1:34941): tablet ace54e4655b0450fa319e57351cc5268 (table test-table [id=b1b387ed9810458fbab2bd9d08a83179]) successfully deleted
I20260925 15:17:51.379299 32086 ts_tablet_manager.cc:1918] T ace54e4655b0450fa319e57351cc5268 P f2261dd3d0664afca6e831ead054741d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:17:51.379516 32086 log.cc:1196] T ace54e4655b0450fa319e57351cc5268 P f2261dd3d0664afca6e831ead054741d: Deleting WAL directory at /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349162370577-12320-0/minicluster-data/ts-1/wal/wals/ace54e4655b0450fa319e57351cc5268
I20260925 15:17:51.380124 32086 ts_tablet_manager.cc:1939] T ace54e4655b0450fa319e57351cc5268 P f2261dd3d0664afca6e831ead054741d: Deleting consensus metadata
I20260925 15:17:51.382016 32058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:51.346325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50142 (local address 127.12.8.1:34941)
0925 15:17:51.346499 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:51.346507 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:51.356006 (+ 9499us) server_negotiation.cc:407] Connection header received
0925 15:17:51.356476 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:51.356482 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:51.356566 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:51.356647 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:17:51.358652 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:51.359790 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:51.363756 (+ 3966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:51.364583 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:51.364731 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:51.375777 (+ 11046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:17:51.375793 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:17:51.375813 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:17:51.375854 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:17:51.378956 (+ 3102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:17:51.379860 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:17:51.379867 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:17:51.379870 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:17:51.379996 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:17:51.380427 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:17:51.380434 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:17:51.380442 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:17:51.380779 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:17:51.381097 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:51.381412 (+ 315us) server_negotiation.cc:299] Negotiation successful
0925 15:17:51.381632 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 15:17:51.382025 31572 catalog_manager.cc:4841] TS f2261dd3d0664afca6e831ead054741d (127.12.8.2:45389): tablet ace54e4655b0450fa319e57351cc5268 (table test-table [id=b1b387ed9810458fbab2bd9d08a83179]) successfully deleted
I20260925 15:17:51.382833 31771 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:17:51.494164 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 31648
I20260925 15:17:51.514107 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 31782
Sep 25 15:17:51 dist-test-slave-p468 krb5kdc[31542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.2@KRBTEST.COM
I20260925 15:17:51.532115 32062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:51.515051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55116 (local address 127.12.8.2:45389)
0925 15:17:51.515255 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:51.515261 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:51.515282 (+ 21us) server_negotiation.cc:407] Connection header received
0925 15:17:51.515434 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:51.515439 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:51.515521 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:51.515634 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:17:51.517040 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:51.518393 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:51.519704 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:51.520443 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:51.520588 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:51.525195 (+ 4607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:17:51.525214 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:17:51.525235 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:17:51.525280 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:17:51.529169 (+ 3889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:17:51.529833 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:17:51.529843 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:17:51.529847 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:17:51.530011 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:17:51.530437 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:17:51.530443 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:17:51.530447 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:17:51.530926 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:17:51.531256 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:51.531454 (+ 198us) server_negotiation.cc:299] Negotiation successful
0925 15:17:51.531686 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 15:17:51.532913 31905 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:17:51.622644 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 31782
I20260925 15:17:51.640216 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 31916
Sep 25 15:17:51 dist-test-slave-p468 krb5kdc[31542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.3@KRBTEST.COM
I20260925 15:17:51.656919 32059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:51.641129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34450 (local address 127.12.8.3:39713)
0925 15:17:51.641335 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:51.641340 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:51.641360 (+ 20us) server_negotiation.cc:407] Connection header received
0925 15:17:51.641529 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:51.641534 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:51.641617 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:51.641761 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:17:51.643116 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:51.644096 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:51.645532 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:51.645992 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:51.646141 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:51.650816 (+ 4675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:17:51.650826 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:17:51.650843 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:17:51.650878 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:17:51.654028 (+ 3150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:17:51.654729 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:17:51.654737 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:17:51.654741 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:17:51.654895 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:17:51.655301 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:17:51.655307 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:17:51.655315 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:17:51.655668 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:17:51.655991 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:51.656328 (+ 337us) server_negotiation.cc:299] Negotiation successful
0925 15:17:51.656553 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 15:17:51.657644 32039 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:17:51.781085 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 31916
I20260925 15:17:51.800001 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 31555
I20260925 15:17:51.801105 31617 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:17:51.917472 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 31555
2026-09-25T15:17:51Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4302 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-25T15:17:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:17:51Z Disabled control of system clock
I20260925 15:17:51.995090 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349162370577-12320-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349162370577-12320-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349162370577-12320-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.62:38279
--webserver_interface=127.12.8.62
--webserver_port=0
--builtin_ntp_servers=127.12.8.20:35527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.62:38279
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 15:17:52.318385 32096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:17:52.318787 32096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:17:52.318943 32096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:17:52.326032 32096 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:17:52.326128 32096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:17:52.326236 32096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:17:52.326315 32096 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:17:52.326388 32096 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:17:52.335346 32096 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:35527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349162370577-12320-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349162370577-12320-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.8.62:38279
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.62:38279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.8.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349162370577-12320-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:17:52.336675 32096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:17:52.338377 32096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:17:52.350267 32101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:17:52.350271 32102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:17:52.350181 32104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:17:52.350994 32096 server_base.cc:1034] running on GCE node
I20260925 15:17:52.351831 32096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:17:52.353215 32096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:17:52.354462 32096 hybrid_clock.cc:648] HybridClock initialized: now 1790349472354416 us; error 58 us; skew 500 ppm
I20260925 15:17:52.354835 32096 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:17:52.357136 32096 webserver.cc:460] Webserver started at http://127.12.8.62:46745/ using document root <none> and password file <none>
I20260925 15:17:52.357817 32096 fs_manager.cc:359] Metadata directory not provided
I20260925 15:17:52.357954 32096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:17:52.358366 32096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:17:52.361266 32096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349162370577-12320-0/minicluster-data/master-0/data/instance:
uuid: "13c3304caf114421a656a1d9d7bc6024"
format_stamp: "Formatted at 2026-09-25 15:17:52 on dist-test-slave-p468"
server_key: "d2d2375d7e3005f663d40cc8d2e26de0"
server_key_iv: "2b8595b45cdae95a576e8e77a5e3842c"
server_key_version: "encryptionkey@0"
I20260925 15:17:52.362066 32096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349162370577-12320-0/minicluster-data/master-0/wal/instance:
uuid: "13c3304caf114421a656a1d9d7bc6024"
format_stamp: "Formatted at 2026-09-25 15:17:52 on dist-test-slave-p468"
server_key: "d2d2375d7e3005f663d40cc8d2e26de0"
server_key_iv: "2b8595b45cdae95a576e8e77a5e3842c"
server_key_version: "encryptionkey@0"
I20260925 15:17:52.367050 32096 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260925 15:17:52.371076 32110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:17:52.372320 32096 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 15:17:52.372512 32096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349162370577-12320-0/minicluster-data/master-0/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349162370577-12320-0/minicluster-data/master-0/wal
uuid: "13c3304caf114421a656a1d9d7bc6024"
format_stamp: "Formatted at 2026-09-25 15:17:52 on dist-test-slave-p468"
server_key: "d2d2375d7e3005f663d40cc8d2e26de0"
server_key_iv: "2b8595b45cdae95a576e8e77a5e3842c"
server_key_version: "encryptionkey@0"
I20260925 15:17:52.372730 32096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349162370577-12320-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349162370577-12320-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349162370577-12320-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:17:52.386888 32096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:17:52.387670 32096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:17:52.387955 32096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:17:52.406327 32096 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.62:38279
I20260925 15:17:52.406294 32161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.62:38279 every 8 connection(s)
I20260925 15:17:52.407711 32096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
I20260925 15:17:52.411356 32162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:17:52.418063 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 32096
I20260925 15:17:52.418275 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349162370577-12320-0/minicluster-data/master-0/wal/instance
I20260925 15:17:52.418852 12320 external_mini_cluster.cc:1428] Setting key f8f81d77541a2fdc49fe26e2f8c847ca
I20260925 15:17:52.422063 32162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13c3304caf114421a656a1d9d7bc6024: Bootstrap starting.
I20260925 15:17:52.425180 32162 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 13c3304caf114421a656a1d9d7bc6024: Neither blocks nor log segments found. Creating new log.
I20260925 15:17:52.426510 32162 log.cc:824] T 00000000000000000000000000000000 P 13c3304caf114421a656a1d9d7bc6024: Log is configured to *not* fsync() on all Append() calls
I20260925 15:17:52.429183 32162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13c3304caf114421a656a1d9d7bc6024: No bootstrap required, opened a new log
I20260925 15:17:52.432142 32165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:52.420625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34020 (local address 127.12.8.62:38279)
0925 15:17:52.421600 (+ 975us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:52.421611 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:52.421651 (+ 40us) server_negotiation.cc:407] Connection header received
0925 15:17:52.423389 (+ 1738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:52.423413 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:52.423846 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:52.424212 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:17:52.424742 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:52.426455 (+ 1713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:52.427817 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:52.428457 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:52.428662 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:52.429855 (+ 1193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:17:52.429862 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:17:52.429877 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:17:52.430453 (+ 576us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:17:52.430526 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:17:52.430546 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:17:52.430686 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:52.430907 (+ 221us) server_negotiation.cc:299] Negotiation successful
0925 15:17:52.431191 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":306,"threads_started":1}
I20260925 15:17:52.437007 32162 raft_consensus.cc:348] T 00000000000000000000000000000000 P 13c3304caf114421a656a1d9d7bc6024 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13c3304caf114421a656a1d9d7bc6024" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 38279 } }
I20260925 15:17:52.437435 32162 raft_consensus.cc:374] T 00000000000000000000000000000000 P 13c3304caf114421a656a1d9d7bc6024 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:17:52.437553 32162 raft_consensus.cc:729] T 00000000000000000000000000000000 P 13c3304caf114421a656a1d9d7bc6024 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13c3304caf114421a656a1d9d7bc6024, State: Initialized, Role: FOLLOWER
I20260925 15:17:52.438179 32162 consensus_queue.cc:260] T 00000000000000000000000000000000 P 13c3304caf114421a656a1d9d7bc6024 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13c3304caf114421a656a1d9d7bc6024" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 38279 } }
I20260925 15:17:52.438452 32162 raft_consensus.cc:388] T 00000000000000000000000000000000 P 13c3304caf114421a656a1d9d7bc6024 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:17:52.438571 32162 raft_consensus.cc:482] T 00000000000000000000000000000000 P 13c3304caf114421a656a1d9d7bc6024 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:17:52.438689 32162 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 13c3304caf114421a656a1d9d7bc6024 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:17:52.440709 32162 raft_consensus.cc:504] T 00000000000000000000000000000000 P 13c3304caf114421a656a1d9d7bc6024 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13c3304caf114421a656a1d9d7bc6024" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 38279 } }
I20260925 15:17:52.441141 32162 leader_election.cc:302] T 00000000000000000000000000000000 P 13c3304caf114421a656a1d9d7bc6024 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 13c3304caf114421a656a1d9d7bc6024; no voters:
I20260925 15:17:52.441684 32162 leader_election.cc:288] T 00000000000000000000000000000000 P 13c3304caf114421a656a1d9d7bc6024 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:17:52.441867 32167 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 13c3304caf114421a656a1d9d7bc6024 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:17:52.442637 32167 raft_consensus.cc:686] T 00000000000000000000000000000000 P 13c3304caf114421a656a1d9d7bc6024 [term 1 LEADER]: Becoming Leader. State: Replica: 13c3304caf114421a656a1d9d7bc6024, State: Running, Role: LEADER
I20260925 15:17:52.443137 32167 consensus_queue.cc:237] T 00000000000000000000000000000000 P 13c3304caf114421a656a1d9d7bc6024 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13c3304caf114421a656a1d9d7bc6024" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 38279 } }
I20260925 15:17:52.443851 32162 sys_catalog.cc:564] T 00000000000000000000000000000000 P 13c3304caf114421a656a1d9d7bc6024 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:17:52.445905 32168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13c3304caf114421a656a1d9d7bc6024 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "13c3304caf114421a656a1d9d7bc6024" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13c3304caf114421a656a1d9d7bc6024" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 38279 } } }
I20260925 15:17:52.445940 32169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13c3304caf114421a656a1d9d7bc6024 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 13c3304caf114421a656a1d9d7bc6024. Latest consensus state: current_term: 1 leader_uuid: "13c3304caf114421a656a1d9d7bc6024" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13c3304caf114421a656a1d9d7bc6024" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 38279 } } }
I20260925 15:17:52.446372 32168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13c3304caf114421a656a1d9d7bc6024 [sys.catalog]: This master's current role is: LEADER
I20260925 15:17:52.446440 32169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13c3304caf114421a656a1d9d7bc6024 [sys.catalog]: This master's current role is: LEADER
I20260925 15:17:52.447234 32173 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:17:52.452378 32173 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:17:52.460201 32173 catalog_manager.cc:1347] Generated new cluster ID: 9cb50255044849a8b90012962dd54b13
I20260925 15:17:52.460310 32173 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:17:52.479635 32173 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:17:52.481472 32173 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:17:52.495869 32173 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 13c3304caf114421a656a1d9d7bc6024: Generated new TSK 0
I20260925 15:17:52.496976 32173 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:17:52.506295 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349162370577-12320-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349162370577-12320-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349162370577-12320-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349162370577-12320-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.1:0
--local_ip_for_outbound_sockets=127.12.8.1
--webserver_interface=127.12.8.1
--webserver_port=0
--tserver_master_addrs=127.12.8.62:38279
--builtin_ntp_servers=127.12.8.20:35527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 15:17:52.834105 32186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:17:52.834468 32186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:17:52.834653 32186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:17:52.841938 32186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:17:52.842088 32186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:17:52.842293 32186 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.1
I20260925 15:17:52.851715 32186 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:35527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349162370577-12320-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349162370577-12320-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349162370577-12320-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.8.1
--webserver_port=0
--tserver_master_addrs=127.12.8.62:38279
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.1
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349162370577-12320-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:17:52.853159 32186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:17:52.854991 32186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:17:52.868520 32194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:17:52.868644 32192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:17:52.868834 32191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:17:52.869625 32186 server_base.cc:1034] running on GCE node
I20260925 15:17:52.870361 32186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:17:52.871380 32186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:17:52.872648 32186 hybrid_clock.cc:648] HybridClock initialized: now 1790349472872583 us; error 49 us; skew 500 ppm
I20260925 15:17:52.873001 32186 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:17:52.875397 32186 webserver.cc:460] Webserver started at http://127.12.8.1:33275/ using document root <none> and password file <none>
I20260925 15:17:52.876216 32186 fs_manager.cc:359] Metadata directory not provided
I20260925 15:17:52.876360 32186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:17:52.876731 32186 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:17:52.880402 32186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349162370577-12320-0/minicluster-data/ts-0/data/instance:
uuid: "50719f6cc39f45f5a35545528fa16a05"
format_stamp: "Formatted at 2026-09-25 15:17:52 on dist-test-slave-p468"
server_key: "ad7efe3b9f157e0fe4597216f817ca08"
server_key_iv: "013db5c2173225c25681eca74055e7f4"
server_key_version: "encryptionkey@0"
I20260925 15:17:52.881217 32186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349162370577-12320-0/minicluster-data/ts-0/wal/instance:
uuid: "50719f6cc39f45f5a35545528fa16a05"
format_stamp: "Formatted at 2026-09-25 15:17:52 on dist-test-slave-p468"
server_key: "ad7efe3b9f157e0fe4597216f817ca08"
server_key_iv: "013db5c2173225c25681eca74055e7f4"
server_key_version: "encryptionkey@0"
I20260925 15:17:52.886300 32186 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260925 15:17:52.889597 32200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:17:52.891029 32186 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260925 15:17:52.891232 32186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349162370577-12320-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349162370577-12320-0/minicluster-data/ts-0/wal
uuid: "50719f6cc39f45f5a35545528fa16a05"
format_stamp: "Formatted at 2026-09-25 15:17:52 on dist-test-slave-p468"
server_key: "ad7efe3b9f157e0fe4597216f817ca08"
server_key_iv: "013db5c2173225c25681eca74055e7f4"
server_key_version: "encryptionkey@0"
I20260925 15:17:52.891474 32186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349162370577-12320-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349162370577-12320-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349162370577-12320-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:17:52.935853 32186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:17:52.936769 32186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:17:52.937073 32186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:17:52.938122 32186 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:17:52.939751 32186 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:17:52.939888 32186 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:17:52.940034 32186 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:17:52.940140 32186 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:17:52.974174 32186 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.1:35491
I20260925 15:17:52.974292 32312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.1:35491 every 8 connection(s)
I20260925 15:17:52.975524 32186 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349162370577-12320-0/minicluster-data/ts-0/data/info.pb
I20260925 15:17:52.981102 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 32186
I20260925 15:17:52.981272 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349162370577-12320-0/minicluster-data/ts-0/wal/instance
I20260925 15:17:52.981706 12320 external_mini_cluster.cc:1428] Setting key 8754d411b53f5425ce73583cd23de022
I20260925 15:17:52.984750 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349162370577-12320-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349162370577-12320-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349162370577-12320-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349162370577-12320-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.2:0
--local_ip_for_outbound_sockets=127.12.8.2
--webserver_interface=127.12.8.2
--webserver_port=0
--tserver_master_addrs=127.12.8.62:38279
--builtin_ntp_servers=127.12.8.20:35527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 15:17:52.989779 32315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:52.978311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.1:54861 (local address 127.12.8.62:38279)
0925 15:17:52.978790 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:52.978796 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:52.980003 (+ 1207us) server_negotiation.cc:407] Connection header received
0925 15:17:52.982090 (+ 2087us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:52.982096 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:52.982184 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:52.982329 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:17:52.983629 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:52.984685 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:52.986514 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:52.987228 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:52.987368 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:52.988786 (+ 1418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:17:52.988794 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:17:52.988802 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:17:52.988825 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:17:52.988878 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:17:52.988896 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:17:52.988992 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:52.989223 (+ 231us) server_negotiation.cc:299] Negotiation successful
0925 15:17:52.989331 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":228,"threads_started":1}
I20260925 15:17:52.990726 32316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:52.978910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:38279 (local address 127.12.8.1:54861)
0925 15:17:52.979834 (+ 924us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:52.979876 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 15:17:52.981810 (+ 1934us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:17:52.982568 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:17:52.982576 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:17:52.982664 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:17:52.983457 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:52.983477 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:52.984901 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:52.984910 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:52.986368 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:52.986381 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:52.987435 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:52.987440 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:52.987667 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:52.988437 (+ 770us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:17:52.988440 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:17:52.988487 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:17:52.988576 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:17:52.989001 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:17:52.989009 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:17:52.989015 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 15:17:52.989183 (+ 168us) client_negotiation.cc:240] Negotiation successful
0925 15:17:52.989476 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":273,"threads_started":1}
I20260925 15:17:52.992352 32313 heartbeater.cc:347] Connected to a master server at 127.12.8.62:38279
I20260925 15:17:52.992677 32313 heartbeater.cc:464] Registering TS with master...
I20260925 15:17:52.993422 32313 heartbeater.cc:511] Master 127.12.8.62:38279 requested a full tablet report, sending...
I20260925 15:17:52.995896 32127 ts_manager.cc:194] Registered new tserver with Master: 50719f6cc39f45f5a35545528fa16a05 (127.12.8.1:35491)
I20260925 15:17:52.997447 32127 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.8.1:54861
W20260925 15:17:53.312073 32317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:17:53.312404 32317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:17:53.312578 32317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:17:53.319905 32317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:17:53.320058 32317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:17:53.320253 32317 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.2
I20260925 15:17:53.329424 32317 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:35527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349162370577-12320-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349162370577-12320-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349162370577-12320-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.8.2
--webserver_port=0
--tserver_master_addrs=127.12.8.62:38279
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.2
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349162370577-12320-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:17:53.330817 32317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:17:53.332576 32317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:17:53.346311 32322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:17:53.346274 32323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:17:53.346232 32325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:17:53.348789 32317 server_base.cc:1034] running on GCE node
I20260925 15:17:53.349516 32317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:17:53.350649 32317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:17:53.351814 32317 hybrid_clock.cc:648] HybridClock initialized: now 1790349473351752 us; error 54 us; skew 500 ppm
I20260925 15:17:53.352191 32317 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:17:53.354650 32317 webserver.cc:460] Webserver started at http://127.12.8.2:35827/ using document root <none> and password file <none>
I20260925 15:17:53.355341 32317 fs_manager.cc:359] Metadata directory not provided
I20260925 15:17:53.355428 32317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:17:53.355787 32317 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:17:53.358744 32317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349162370577-12320-0/minicluster-data/ts-1/data/instance:
uuid: "faad434d972f4ce394cce46d5462b99c"
format_stamp: "Formatted at 2026-09-25 15:17:53 on dist-test-slave-p468"
server_key: "aef3b5d9dd3b9229eb6719358464fab7"
server_key_iv: "e3adc7894971a69a1ffa39b25fc3d84a"
server_key_version: "encryptionkey@0"
I20260925 15:17:53.359536 32317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349162370577-12320-0/minicluster-data/ts-1/wal/instance:
uuid: "faad434d972f4ce394cce46d5462b99c"
format_stamp: "Formatted at 2026-09-25 15:17:53 on dist-test-slave-p468"
server_key: "aef3b5d9dd3b9229eb6719358464fab7"
server_key_iv: "e3adc7894971a69a1ffa39b25fc3d84a"
server_key_version: "encryptionkey@0"
I20260925 15:17:53.364689 32317 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260925 15:17:53.368176 32331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:17:53.369654 32317 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 15:17:53.369887 32317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349162370577-12320-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349162370577-12320-0/minicluster-data/ts-1/wal
uuid: "faad434d972f4ce394cce46d5462b99c"
format_stamp: "Formatted at 2026-09-25 15:17:53 on dist-test-slave-p468"
server_key: "aef3b5d9dd3b9229eb6719358464fab7"
server_key_iv: "e3adc7894971a69a1ffa39b25fc3d84a"
server_key_version: "encryptionkey@0"
I20260925 15:17:53.370133 32317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349162370577-12320-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349162370577-12320-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349162370577-12320-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:17:53.392735 32317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:17:53.393522 32317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:17:53.393810 32317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:17:53.394850 32317 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:17:53.396402 32317 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:17:53.396523 32317 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:17:53.396651 32317 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:17:53.396749 32317 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:17:53.436205 32317 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.2:39009
I20260925 15:17:53.436301 32443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.2:39009 every 8 connection(s)
I20260925 15:17:53.437446 32317 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349162370577-12320-0/minicluster-data/ts-1/data/info.pb
I20260925 15:17:53.441365 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 32317
I20260925 15:17:53.441545 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349162370577-12320-0/minicluster-data/ts-1/wal/instance
I20260925 15:17:53.441984 12320 external_mini_cluster.cc:1428] Setting key 84d99ff3f711b803c14d331fae4ed09d
I20260925 15:17:53.445039 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349162370577-12320-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349162370577-12320-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349162370577-12320-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349162370577-12320-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.8.3:0
--local_ip_for_outbound_sockets=127.12.8.3
--webserver_interface=127.12.8.3
--webserver_port=0
--tserver_master_addrs=127.12.8.62:38279
--builtin_ntp_servers=127.12.8.20:35527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 15:17:53.452536 32315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:53.440284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.2:42959 (local address 127.12.8.62:38279)
0925 15:17:53.440505 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:53.440512 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:53.442161 (+ 1649us) server_negotiation.cc:407] Connection header received
0925 15:17:53.444586 (+ 2425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:53.444592 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:53.444714 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:53.444831 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:17:53.446064 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:53.447264 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:53.448990 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:53.449703 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:53.449847 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:53.451608 (+ 1761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:17:53.451617 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:17:53.451624 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:17:53.451649 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:17:53.451699 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:17:53.451717 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:17:53.451816 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:53.452042 (+ 226us) server_negotiation.cc:299] Negotiation successful
0925 15:17:53.452157 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 15:17:53.453534 32446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:53.440898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:38279 (local address 127.12.8.2:42959)
0925 15:17:53.442005 (+ 1107us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:53.442051 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 15:17:53.444324 (+ 2273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:17:53.445087 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:17:53.445096 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:17:53.445228 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:17:53.445886 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:53.445904 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:53.447437 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:53.447442 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:53.448831 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:53.448844 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:53.450087 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:53.450093 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:53.450513 (+ 420us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:53.451289 (+ 776us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:17:53.451292 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:17:53.451332 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:17:53.451413 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:17:53.451826 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:17:53.451833 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:17:53.451839 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 15:17:53.451981 (+ 142us) client_negotiation.cc:240] Negotiation successful
0925 15:17:53.452353 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":356,"threads_started":1}
I20260925 15:17:53.455219 32444 heartbeater.cc:347] Connected to a master server at 127.12.8.62:38279
I20260925 15:17:53.455564 32444 heartbeater.cc:464] Registering TS with master...
I20260925 15:17:53.456364 32444 heartbeater.cc:511] Master 127.12.8.62:38279 requested a full tablet report, sending...
I20260925 15:17:53.457914 32127 ts_manager.cc:194] Registered new tserver with Master: faad434d972f4ce394cce46d5462b99c (127.12.8.2:39009)
I20260925 15:17:53.459233 32127 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.8.2:42959
W20260925 15:17:53.777818 32447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:17:53.778123 32447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:17:53.778287 32447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:17:53.785960 32447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:17:53.786067 32447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:17:53.786239 32447 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.8.3
I20260925 15:17:53.795219 32447 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:35527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349162370577-12320-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349162370577-12320-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.8.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349162370577-12320-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.8.3
--webserver_port=0
--tserver_master_addrs=127.12.8.62:38279
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.8.3
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349162370577-12320-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:17:53.796482 32447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:17:53.798138 32447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:17:53.811507 32453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:17:53.811544 32455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:17:53.811474 32452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:17:53.812695 32447 server_base.cc:1034] running on GCE node
I20260925 15:17:53.813270 32447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:17:53.814179 32447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:17:53.815456 32447 hybrid_clock.cc:648] HybridClock initialized: now 1790349473815398 us; error 56 us; skew 500 ppm
I20260925 15:17:53.815783 32447 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:17:53.817937 32447 webserver.cc:460] Webserver started at http://127.12.8.3:42493/ using document root <none> and password file <none>
I20260925 15:17:53.818732 32447 fs_manager.cc:359] Metadata directory not provided
I20260925 15:17:53.818866 32447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:17:53.819198 32447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:17:53.822466 32447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349162370577-12320-0/minicluster-data/ts-2/data/instance:
uuid: "24821bc126ca4afdb1d008d349d445de"
format_stamp: "Formatted at 2026-09-25 15:17:53 on dist-test-slave-p468"
server_key: "eb48c9fa5202437ad532acc596c42d3b"
server_key_iv: "efbd708c160c7b9006060a1e98f8a06e"
server_key_version: "encryptionkey@0"
I20260925 15:17:53.823267 32447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349162370577-12320-0/minicluster-data/ts-2/wal/instance:
uuid: "24821bc126ca4afdb1d008d349d445de"
format_stamp: "Formatted at 2026-09-25 15:17:53 on dist-test-slave-p468"
server_key: "eb48c9fa5202437ad532acc596c42d3b"
server_key_iv: "efbd708c160c7b9006060a1e98f8a06e"
server_key_version: "encryptionkey@0"
I20260925 15:17:53.828189 32447 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260925 15:17:53.831535 32461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:17:53.832867 32447 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 15:17:53.833065 32447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349162370577-12320-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349162370577-12320-0/minicluster-data/ts-2/wal
uuid: "24821bc126ca4afdb1d008d349d445de"
format_stamp: "Formatted at 2026-09-25 15:17:53 on dist-test-slave-p468"
server_key: "eb48c9fa5202437ad532acc596c42d3b"
server_key_iv: "efbd708c160c7b9006060a1e98f8a06e"
server_key_version: "encryptionkey@0"
I20260925 15:17:53.833293 32447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349162370577-12320-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349162370577-12320-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349162370577-12320-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:17:53.862591 32447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:17:53.863664 32447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:17:53.863966 32447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:17:53.865303 32447 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:17:53.867120 32447 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:17:53.867208 32447 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:17:53.867342 32447 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:17:53.867419 32447 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:17:53.899911 32447 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.3:45339
I20260925 15:17:53.900008 32573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.3:45339 every 8 connection(s)
I20260925 15:17:53.901268 32447 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349162370577-12320-0/minicluster-data/ts-2/data/info.pb
I20260925 15:17:53.902303 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 32447
I20260925 15:17:53.902510 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349162370577-12320-0/minicluster-data/ts-2/wal/instance
I20260925 15:17:53.903088 12320 external_mini_cluster.cc:1428] Setting key c162e3d078286950ff1886efbcee0711
I20260925 15:17:53.914502 32315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:53.904387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.3:48137 (local address 127.12.8.62:38279)
0925 15:17:53.904568 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:53.904577 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:53.906027 (+ 1450us) server_negotiation.cc:407] Connection header received
0925 15:17:53.907936 (+ 1909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:53.907942 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:53.908034 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:53.908143 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:17:53.909205 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:53.909932 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:53.911551 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:53.911967 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:53.912109 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:53.913505 (+ 1396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:17:53.913512 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:17:53.913517 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:17:53.913541 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:17:53.913574 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:17:53.913588 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:17:53.913706 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:53.913970 (+ 264us) server_negotiation.cc:299] Negotiation successful
0925 15:17:53.914115 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 15:17:53.915628 32576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:53.904940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:38279 (local address 127.12.8.3:48137)
0925 15:17:53.905869 (+ 929us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:53.905913 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 15:17:53.907667 (+ 1754us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:17:53.908382 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:17:53.908391 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:17:53.908482 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:17:53.909031 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:53.909046 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:53.910080 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:53.910086 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:53.911392 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:53.911404 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:53.912196 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:53.912205 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:53.912451 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:53.913125 (+ 674us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:17:53.913128 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:17:53.913175 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:17:53.913257 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:17:53.913704 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:17:53.913717 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:17:53.913728 (+ 11us) client_negotiation.cc:769] Sending connection context
0925 15:17:53.913967 (+ 239us) client_negotiation.cc:240] Negotiation successful
0925 15:17:53.914416 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":311,"threads_started":1}
I20260925 15:17:53.917171 32574 heartbeater.cc:347] Connected to a master server at 127.12.8.62:38279
I20260925 15:17:53.917497 32574 heartbeater.cc:464] Registering TS with master...
I20260925 15:17:53.918346 32574 heartbeater.cc:511] Master 127.12.8.62:38279 requested a full tablet report, sending...
I20260925 15:17:53.919832 32127 ts_manager.cc:194] Registered new tserver with Master: 24821bc126ca4afdb1d008d349d445de (127.12.8.3:45339)
I20260925 15:17:53.920614 32127 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.8.3:48137
I20260925 15:17:53.931087 12320 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:17:53.941602 32315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:53.935242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55294 (local address 127.12.8.62:38279)
0925 15:17:53.935514 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:53.935521 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:53.935830 (+ 309us) server_negotiation.cc:407] Connection header received
0925 15:17:53.936047 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:53.936057 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:53.936131 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:53.936246 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:17:53.936907 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:53.937729 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:53.939284 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:53.939773 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:53.939887 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:53.941013 (+ 1126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:17:53.941018 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:17:53.941022 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:17:53.941039 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:17:53.941084 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:17:53.941096 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:17:53.941175 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:53.941294 (+ 119us) server_negotiation.cc:299] Negotiation successful
0925 15:17:53.941368 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 15:17:53.945653 32127 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55294:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:17:53.952736 32127 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:17:53.978410 32588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:53.964979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.1:35491 (local address 127.0.0.1:49168)
0925 15:17:53.966543 (+ 1564us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:53.966599 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 15:17:53.966802 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:17:53.968720 (+ 1918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:17:53.968730 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:17:53.968771 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:17:53.969328 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:53.969356 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:53.972433 (+ 3077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:53.972441 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:53.975115 (+ 2674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:53.975131 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:53.977344 (+ 2213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:53.977352 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:53.977578 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:53.977633 (+ 55us) client_negotiation.cc:769] Sending connection context
0925 15:17:53.977791 (+ 158us) client_negotiation.cc:240] Negotiation successful
0925 15:17:53.978009 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1178,"mutex_wait_us":67,"spinlock_wait_cycles":3840,"thread_start_us":984,"threads_started":1}
I20260925 15:17:53.978474 32589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:53.965582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49168 (local address 127.12.8.1:35491)
0925 15:17:53.967992 (+ 2410us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:53.968000 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:53.968025 (+ 25us) server_negotiation.cc:407] Connection header received
0925 15:17:53.968127 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:53.968137 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:53.968414 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:53.969651 (+ 1237us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:17:53.969920 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:53.972270 (+ 2350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:53.975357 (+ 3087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:53.977092 (+ 1735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:53.977300 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:53.977685 (+ 385us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:53.977829 (+ 144us) server_negotiation.cc:299] Negotiation successful
0925 15:17:53.978003 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2275,"thread_start_us":2181,"threads_started":1}
I20260925 15:17:53.980082 32587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:53.965015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.3:45339 (local address 127.0.0.1:59756)
0925 15:17:53.967422 (+ 2407us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:53.967446 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 15:17:53.967556 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:17:53.968839 (+ 1283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:17:53.968851 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:17:53.968915 (+ 64us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:17:53.969491 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:53.969506 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:53.972165 (+ 2659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:53.972172 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:53.974973 (+ 2801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:53.974991 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:53.979239 (+ 4248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:53.979252 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:53.979444 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:53.979478 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 15:17:53.979572 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 15:17:53.979711 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2277,"mutex_wait_us":3,"spinlock_wait_cycles":2432,"thread_start_us":2133,"threads_started":1}
I20260925 15:17:53.980865 32586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:53.965016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59756 (local address 127.12.8.3:45339)
0925 15:17:53.965687 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:53.965698 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:53.967733 (+ 2035us) server_negotiation.cc:407] Connection header received
0925 15:17:53.967921 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:53.967931 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:53.968272 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:53.968493 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:17:53.970390 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:53.972029 (+ 1639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:53.975252 (+ 3223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:53.976726 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:53.976878 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:53.977166 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:53.980337 (+ 3171us) server_negotiation.cc:299] Negotiation successful
0925 15:17:53.980534 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":371,"threads_started":1}
I20260925 15:17:53.982367 32591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:53.968083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.2:39009 (local address 127.0.0.1:43242)
0925 15:17:53.971504 (+ 3421us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:53.971528 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 15:17:53.971651 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:17:53.972313 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:17:53.972320 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:17:53.972358 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:17:53.973032 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:53.973056 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:53.976499 (+ 3443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:53.976504 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:53.979626 (+ 3122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:53.979644 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:53.981539 (+ 1895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:53.981547 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:53.981731 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:53.981774 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 15:17:53.981868 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 15:17:53.981995 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3317,"mutex_wait_us":3,"thread_start_us":3200,"threads_started":1}
I20260925 15:17:53.982378 32590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:53.967806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43242 (local address 127.12.8.2:39009)
0925 15:17:53.971254 (+ 3448us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:53.971262 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:53.971643 (+ 381us) server_negotiation.cc:407] Connection header received
0925 15:17:53.971810 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:53.971819 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:53.972087 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:53.972297 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:17:53.973277 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:53.976323 (+ 3046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:53.979862 (+ 3539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:53.981301 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:53.981435 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:53.981667 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:53.981919 (+ 252us) server_negotiation.cc:299] Negotiation successful
0925 15:17:53.982038 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3280,"thread_start_us":3171,"threads_started":1}
I20260925 15:17:53.983448 32248 tablet_service.cc:1467] Processing CreateTablet for tablet 7dc4693a133c47b58477e5c1a9b6456a (DEFAULT_TABLE table=test-table [id=8907cb03e3b44c4eaa82b10f852ce7c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:17:53.985052 32248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dc4693a133c47b58477e5c1a9b6456a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:17:53.985055 32509 tablet_service.cc:1467] Processing CreateTablet for tablet 7dc4693a133c47b58477e5c1a9b6456a (DEFAULT_TABLE table=test-table [id=8907cb03e3b44c4eaa82b10f852ce7c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:17:53.985760 32379 tablet_service.cc:1467] Processing CreateTablet for tablet 7dc4693a133c47b58477e5c1a9b6456a (DEFAULT_TABLE table=test-table [id=8907cb03e3b44c4eaa82b10f852ce7c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:17:53.986490 32509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dc4693a133c47b58477e5c1a9b6456a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:17:53.987023 32379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dc4693a133c47b58477e5c1a9b6456a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:17:53.996632 32592 tablet_bootstrap.cc:492] T 7dc4693a133c47b58477e5c1a9b6456a P 50719f6cc39f45f5a35545528fa16a05: Bootstrap starting.
I20260925 15:17:54.000321 32313 heartbeater.cc:503] Master 127.12.8.62:38279 was elected leader, sending a full tablet report...
I20260925 15:17:54.000705 32592 tablet_bootstrap.cc:654] T 7dc4693a133c47b58477e5c1a9b6456a P 50719f6cc39f45f5a35545528fa16a05: Neither blocks nor log segments found. Creating new log.
I20260925 15:17:54.002022 32592 log.cc:824] T 7dc4693a133c47b58477e5c1a9b6456a P 50719f6cc39f45f5a35545528fa16a05: Log is configured to *not* fsync() on all Append() calls
I20260925 15:17:54.008049 32593 tablet_bootstrap.cc:492] T 7dc4693a133c47b58477e5c1a9b6456a P faad434d972f4ce394cce46d5462b99c: Bootstrap starting.
I20260925 15:17:54.008224 32594 tablet_bootstrap.cc:492] T 7dc4693a133c47b58477e5c1a9b6456a P 24821bc126ca4afdb1d008d349d445de: Bootstrap starting.
I20260925 15:17:54.011590 32594 tablet_bootstrap.cc:654] T 7dc4693a133c47b58477e5c1a9b6456a P 24821bc126ca4afdb1d008d349d445de: Neither blocks nor log segments found. Creating new log.
I20260925 15:17:54.012125 32593 tablet_bootstrap.cc:654] T 7dc4693a133c47b58477e5c1a9b6456a P faad434d972f4ce394cce46d5462b99c: Neither blocks nor log segments found. Creating new log.
I20260925 15:17:54.012667 32594 log.cc:824] T 7dc4693a133c47b58477e5c1a9b6456a P 24821bc126ca4afdb1d008d349d445de: Log is configured to *not* fsync() on all Append() calls
I20260925 15:17:54.013196 32592 tablet_bootstrap.cc:492] T 7dc4693a133c47b58477e5c1a9b6456a P 50719f6cc39f45f5a35545528fa16a05: No bootstrap required, opened a new log
I20260925 15:17:54.013352 32593 log.cc:824] T 7dc4693a133c47b58477e5c1a9b6456a P faad434d972f4ce394cce46d5462b99c: Log is configured to *not* fsync() on all Append() calls
I20260925 15:17:54.013550 32592 ts_tablet_manager.cc:1397] T 7dc4693a133c47b58477e5c1a9b6456a P 50719f6cc39f45f5a35545528fa16a05: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.008s
I20260925 15:17:54.015394 32594 tablet_bootstrap.cc:492] T 7dc4693a133c47b58477e5c1a9b6456a P 24821bc126ca4afdb1d008d349d445de: No bootstrap required, opened a new log
I20260925 15:17:54.015699 32594 ts_tablet_manager.cc:1397] T 7dc4693a133c47b58477e5c1a9b6456a P 24821bc126ca4afdb1d008d349d445de: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260925 15:17:54.017856 32593 tablet_bootstrap.cc:492] T 7dc4693a133c47b58477e5c1a9b6456a P faad434d972f4ce394cce46d5462b99c: No bootstrap required, opened a new log
I20260925 15:17:54.018275 32593 ts_tablet_manager.cc:1397] T 7dc4693a133c47b58477e5c1a9b6456a P faad434d972f4ce394cce46d5462b99c: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.006s
I20260925 15:17:54.023216 32594 raft_consensus.cc:348] T 7dc4693a133c47b58477e5c1a9b6456a P 24821bc126ca4afdb1d008d349d445de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24821bc126ca4afdb1d008d349d445de" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 45339 } } peers { permanent_uuid: "50719f6cc39f45f5a35545528fa16a05" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 35491 } } peers { permanent_uuid: "faad434d972f4ce394cce46d5462b99c" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 39009 } }
I20260925 15:17:54.023603 32594 raft_consensus.cc:374] T 7dc4693a133c47b58477e5c1a9b6456a P 24821bc126ca4afdb1d008d349d445de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:17:54.023744 32594 raft_consensus.cc:729] T 7dc4693a133c47b58477e5c1a9b6456a P 24821bc126ca4afdb1d008d349d445de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24821bc126ca4afdb1d008d349d445de, State: Initialized, Role: FOLLOWER
I20260925 15:17:54.023669 32592 raft_consensus.cc:348] T 7dc4693a133c47b58477e5c1a9b6456a P 50719f6cc39f45f5a35545528fa16a05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24821bc126ca4afdb1d008d349d445de" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 45339 } } peers { permanent_uuid: "50719f6cc39f45f5a35545528fa16a05" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 35491 } } peers { permanent_uuid: "faad434d972f4ce394cce46d5462b99c" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 39009 } }
I20260925 15:17:54.024194 32592 raft_consensus.cc:374] T 7dc4693a133c47b58477e5c1a9b6456a P 50719f6cc39f45f5a35545528fa16a05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:17:54.024300 32592 raft_consensus.cc:729] T 7dc4693a133c47b58477e5c1a9b6456a P 50719f6cc39f45f5a35545528fa16a05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50719f6cc39f45f5a35545528fa16a05, State: Initialized, Role: FOLLOWER
I20260925 15:17:54.024423 32594 consensus_queue.cc:260] T 7dc4693a133c47b58477e5c1a9b6456a P 24821bc126ca4afdb1d008d349d445de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24821bc126ca4afdb1d008d349d445de" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 45339 } } peers { permanent_uuid: "50719f6cc39f45f5a35545528fa16a05" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 35491 } } peers { permanent_uuid: "faad434d972f4ce394cce46d5462b99c" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 39009 } }
I20260925 15:17:54.024899 32592 consensus_queue.cc:260] T 7dc4693a133c47b58477e5c1a9b6456a P 50719f6cc39f45f5a35545528fa16a05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24821bc126ca4afdb1d008d349d445de" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 45339 } } peers { permanent_uuid: "50719f6cc39f45f5a35545528fa16a05" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 35491 } } peers { permanent_uuid: "faad434d972f4ce394cce46d5462b99c" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 39009 } }
I20260925 15:17:54.025601 32574 heartbeater.cc:503] Master 127.12.8.62:38279 was elected leader, sending a full tablet report...
I20260925 15:17:54.026377 32594 ts_tablet_manager.cc:1428] T 7dc4693a133c47b58477e5c1a9b6456a P 24821bc126ca4afdb1d008d349d445de: Time spent starting tablet: real 0.011s user 0.005s sys 0.005s
I20260925 15:17:54.030079 32593 raft_consensus.cc:348] T 7dc4693a133c47b58477e5c1a9b6456a P faad434d972f4ce394cce46d5462b99c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24821bc126ca4afdb1d008d349d445de" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 45339 } } peers { permanent_uuid: "50719f6cc39f45f5a35545528fa16a05" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 35491 } } peers { permanent_uuid: "faad434d972f4ce394cce46d5462b99c" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 39009 } }
I20260925 15:17:54.030738 32593 raft_consensus.cc:374] T 7dc4693a133c47b58477e5c1a9b6456a P faad434d972f4ce394cce46d5462b99c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:17:54.031597 32593 raft_consensus.cc:729] T 7dc4693a133c47b58477e5c1a9b6456a P faad434d972f4ce394cce46d5462b99c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: faad434d972f4ce394cce46d5462b99c, State: Initialized, Role: FOLLOWER
I20260925 15:17:54.031795 32592 ts_tablet_manager.cc:1428] T 7dc4693a133c47b58477e5c1a9b6456a P 50719f6cc39f45f5a35545528fa16a05: Time spent starting tablet: real 0.018s user 0.017s sys 0.000s
I20260925 15:17:54.032338 32593 consensus_queue.cc:260] T 7dc4693a133c47b58477e5c1a9b6456a P faad434d972f4ce394cce46d5462b99c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24821bc126ca4afdb1d008d349d445de" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 45339 } } peers { permanent_uuid: "50719f6cc39f45f5a35545528fa16a05" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 35491 } } peers { permanent_uuid: "faad434d972f4ce394cce46d5462b99c" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 39009 } }
I20260925 15:17:54.033485 32444 heartbeater.cc:503] Master 127.12.8.62:38279 was elected leader, sending a full tablet report...
I20260925 15:17:54.034121 32593 ts_tablet_manager.cc:1428] T 7dc4693a133c47b58477e5c1a9b6456a P faad434d972f4ce394cce46d5462b99c: Time spent starting tablet: real 0.016s user 0.013s sys 0.001s
W20260925 15:17:54.153250 32575 tablet.cc:2369] T 7dc4693a133c47b58477e5c1a9b6456a P 24821bc126ca4afdb1d008d349d445de: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:17:54.166322 32598 raft_consensus.cc:482] T 7dc4693a133c47b58477e5c1a9b6456a P 24821bc126ca4afdb1d008d349d445de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:17:54.166700 32598 raft_consensus.cc:504] T 7dc4693a133c47b58477e5c1a9b6456a P 24821bc126ca4afdb1d008d349d445de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24821bc126ca4afdb1d008d349d445de" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 45339 } } peers { permanent_uuid: "50719f6cc39f45f5a35545528fa16a05" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 35491 } } peers { permanent_uuid: "faad434d972f4ce394cce46d5462b99c" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 39009 } }
I20260925 15:17:54.168958 32598 leader_election.cc:288] T 7dc4693a133c47b58477e5c1a9b6456a P 24821bc126ca4afdb1d008d349d445de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 50719f6cc39f45f5a35545528fa16a05 (127.12.8.1:35491), faad434d972f4ce394cce46d5462b99c (127.12.8.2:39009)
I20260925 15:17:54.176231 32590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:54.169498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.3:42033 (local address 127.12.8.2:39009)
0925 15:17:54.169726 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:54.169734 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:54.169757 (+ 23us) server_negotiation.cc:407] Connection header received
0925 15:17:54.169921 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:54.169928 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:54.170018 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:54.170121 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:17:54.170742 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:54.171883 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:54.174397 (+ 2514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:54.175228 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:54.175337 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:54.175403 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:54.175822 (+ 419us) server_negotiation.cc:299] Negotiation successful
0925 15:17:54.175955 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 15:17:54.176282 32576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:54.169226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.2:39009 (local address 127.12.8.3:42033)
0925 15:17:54.169569 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:54.169593 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 15:17:54.169737 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:17:54.170356 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:17:54.170359 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:17:54.170380 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:17:54.170591 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:54.170603 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:54.172014 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:54.172017 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:54.174145 (+ 2128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:54.174161 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:54.175400 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:54.175408 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:54.175633 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:54.175683 (+ 50us) client_negotiation.cc:769] Sending connection context
0925 15:17:54.175783 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 15:17:54.175980 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181}
I20260925 15:17:54.176653 32601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:54.169303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.1:35491 (local address 127.12.8.3:42553)
0925 15:17:54.170017 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:54.170039 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 15:17:54.170150 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:17:54.170508 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:17:54.170517 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:17:54.170540 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:17:54.170878 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:54.170887 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:54.172310 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:54.172315 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:54.174368 (+ 2053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:17:54.174381 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:17:54.175844 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:17:54.175850 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:17:54.176090 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:54.176128 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 15:17:54.176222 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 15:17:54.176370 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":744,"threads_started":1}
I20260925 15:17:54.176801 32589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:54.169939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.8.3:42553 (local address 127.12.8.1:35491)
0925 15:17:54.170162 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:54.170168 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:54.170212 (+ 44us) server_negotiation.cc:407] Connection header received
0925 15:17:54.170294 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:54.170303 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:54.170392 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:54.170471 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:17:54.171048 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:54.172159 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:54.174575 (+ 2416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:54.175625 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:54.175779 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:54.175880 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:54.176381 (+ 501us) server_negotiation.cc:299] Negotiation successful
0925 15:17:54.176537 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 15:17:54.177482 32399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dc4693a133c47b58477e5c1a9b6456a" candidate_uuid: "24821bc126ca4afdb1d008d349d445de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "faad434d972f4ce394cce46d5462b99c" is_pre_election: true
I20260925 15:17:54.177977 32399 raft_consensus.cc:2454] T 7dc4693a133c47b58477e5c1a9b6456a P faad434d972f4ce394cce46d5462b99c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24821bc126ca4afdb1d008d349d445de in term 0.
I20260925 15:17:54.177958 32268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dc4693a133c47b58477e5c1a9b6456a" candidate_uuid: "24821bc126ca4afdb1d008d349d445de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50719f6cc39f45f5a35545528fa16a05" is_pre_election: true
I20260925 15:17:54.178609 32268 raft_consensus.cc:2454] T 7dc4693a133c47b58477e5c1a9b6456a P 50719f6cc39f45f5a35545528fa16a05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24821bc126ca4afdb1d008d349d445de in term 0.
I20260925 15:17:54.178637 32462 leader_election.cc:302] T 7dc4693a133c47b58477e5c1a9b6456a P 24821bc126ca4afdb1d008d349d445de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24821bc126ca4afdb1d008d349d445de, faad434d972f4ce394cce46d5462b99c; no voters:
I20260925 15:17:54.178998 32598 raft_consensus.cc:2781] T 7dc4693a133c47b58477e5c1a9b6456a P 24821bc126ca4afdb1d008d349d445de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:17:54.179165 32598 raft_consensus.cc:482] T 7dc4693a133c47b58477e5c1a9b6456a P 24821bc126ca4afdb1d008d349d445de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:17:54.179283 32598 raft_consensus.cc:3037] T 7dc4693a133c47b58477e5c1a9b6456a P 24821bc126ca4afdb1d008d349d445de [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:17:54.181461 32598 raft_consensus.cc:504] T 7dc4693a133c47b58477e5c1a9b6456a P 24821bc126ca4afdb1d008d349d445de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24821bc126ca4afdb1d008d349d445de" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 45339 } } peers { permanent_uuid: "50719f6cc39f45f5a35545528fa16a05" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 35491 } } peers { permanent_uuid: "faad434d972f4ce394cce46d5462b99c" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 39009 } }
I20260925 15:17:54.182076 32598 leader_election.cc:288] T 7dc4693a133c47b58477e5c1a9b6456a P 24821bc126ca4afdb1d008d349d445de [CANDIDATE]: Term 1 election: Requested vote from peers 50719f6cc39f45f5a35545528fa16a05 (127.12.8.1:35491), faad434d972f4ce394cce46d5462b99c (127.12.8.2:39009)
I20260925 15:17:54.182546 32399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dc4693a133c47b58477e5c1a9b6456a" candidate_uuid: "24821bc126ca4afdb1d008d349d445de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "faad434d972f4ce394cce46d5462b99c"
I20260925 15:17:54.182554 32268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dc4693a133c47b58477e5c1a9b6456a" candidate_uuid: "24821bc126ca4afdb1d008d349d445de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50719f6cc39f45f5a35545528fa16a05"
I20260925 15:17:54.182754 32399 raft_consensus.cc:3037] T 7dc4693a133c47b58477e5c1a9b6456a P faad434d972f4ce394cce46d5462b99c [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:17:54.182760 32268 raft_consensus.cc:3037] T 7dc4693a133c47b58477e5c1a9b6456a P 50719f6cc39f45f5a35545528fa16a05 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:17:54.184875 32268 raft_consensus.cc:2454] T 7dc4693a133c47b58477e5c1a9b6456a P 50719f6cc39f45f5a35545528fa16a05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24821bc126ca4afdb1d008d349d445de in term 1.
I20260925 15:17:54.184881 32399 raft_consensus.cc:2454] T 7dc4693a133c47b58477e5c1a9b6456a P faad434d972f4ce394cce46d5462b99c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24821bc126ca4afdb1d008d349d445de in term 1.
I20260925 15:17:54.185279 32463 leader_election.cc:302] T 7dc4693a133c47b58477e5c1a9b6456a P 24821bc126ca4afdb1d008d349d445de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24821bc126ca4afdb1d008d349d445de, 50719f6cc39f45f5a35545528fa16a05; no voters:
I20260925 15:17:54.185627 32598 raft_consensus.cc:2781] T 7dc4693a133c47b58477e5c1a9b6456a P 24821bc126ca4afdb1d008d349d445de [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:17:54.186012 32598 raft_consensus.cc:686] T 7dc4693a133c47b58477e5c1a9b6456a P 24821bc126ca4afdb1d008d349d445de [term 1 LEADER]: Becoming Leader. State: Replica: 24821bc126ca4afdb1d008d349d445de, State: Running, Role: LEADER
I20260925 15:17:54.186480 32598 consensus_queue.cc:237] T 7dc4693a133c47b58477e5c1a9b6456a P 24821bc126ca4afdb1d008d349d445de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24821bc126ca4afdb1d008d349d445de" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 45339 } } peers { permanent_uuid: "50719f6cc39f45f5a35545528fa16a05" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 35491 } } peers { permanent_uuid: "faad434d972f4ce394cce46d5462b99c" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 39009 } }
W20260925 15:17:54.189639 32445 tablet.cc:2369] T 7dc4693a133c47b58477e5c1a9b6456a P faad434d972f4ce394cce46d5462b99c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:17:54.191059 32127 catalog_manager.cc:5495] T 7dc4693a133c47b58477e5c1a9b6456a P 24821bc126ca4afdb1d008d349d445de reported cstate change: term changed from 0 to 1, leader changed from <none> to 24821bc126ca4afdb1d008d349d445de (127.12.8.3). New cstate: current_term: 1 leader_uuid: "24821bc126ca4afdb1d008d349d445de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24821bc126ca4afdb1d008d349d445de" member_type: VOTER last_known_addr { host: "127.12.8.3" port: 45339 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "50719f6cc39f45f5a35545528fa16a05" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 35491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "faad434d972f4ce394cce46d5462b99c" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 39009 } health_report { overall_health: UNKNOWN } } }
I20260925 15:17:54.216209 32586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:54.210894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59772 (local address 127.12.8.3:45339)
0925 15:17:54.211102 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:54.211108 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:54.211128 (+ 20us) server_negotiation.cc:407] Connection header received
0925 15:17:54.211328 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:54.211332 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:54.211408 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:54.211564 (+ 156us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:17:54.211936 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:54.212647 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:54.214044 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:54.214530 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:54.214673 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:54.215152 (+ 479us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:17:54.215648 (+ 496us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:17:54.215826 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:54.215877 (+ 51us) server_negotiation.cc:299] Negotiation successful
0925 15:17:54.215955 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 15:17:54.225042 32268 raft_consensus.cc:1261] T 7dc4693a133c47b58477e5c1a9b6456a P 50719f6cc39f45f5a35545528fa16a05 [term 1 FOLLOWER]: Refusing update from remote peer 24821bc126ca4afdb1d008d349d445de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:17:54.225113 32399 raft_consensus.cc:1261] T 7dc4693a133c47b58477e5c1a9b6456a P faad434d972f4ce394cce46d5462b99c [term 1 FOLLOWER]: Refusing update from remote peer 24821bc126ca4afdb1d008d349d445de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:17:54.226256 32602 consensus_queue.cc:1035] T 7dc4693a133c47b58477e5c1a9b6456a P 24821bc126ca4afdb1d008d349d445de [LEADER]: Connected to new peer: Peer: permanent_uuid: "faad434d972f4ce394cce46d5462b99c" member_type: VOTER last_known_addr { host: "127.12.8.2" port: 39009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:17:54.226678 32598 consensus_queue.cc:1035] T 7dc4693a133c47b58477e5c1a9b6456a P 24821bc126ca4afdb1d008d349d445de [LEADER]: Connected to new peer: Peer: permanent_uuid: "50719f6cc39f45f5a35545528fa16a05" member_type: VOTER last_known_addr { host: "127.12.8.1" port: 35491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 15:17:54.227954 32314 tablet.cc:2369] T 7dc4693a133c47b58477e5c1a9b6456a P 50719f6cc39f45f5a35545528fa16a05: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:17:54.243153 32607 mvcc.cc:205] Tried to move back new op lower bound from 7333271446414258176 to 7333271446271078400. Current Snapshot: MvccSnapshot[applied={T|T < 7333271446414258176}]
I20260925 15:17:54.246594 32606 mvcc.cc:205] Tried to move back new op lower bound from 7333271446414258176 to 7333271446271078400. Current Snapshot: MvccSnapshot[applied={T|T < 7333271446414258176}]
I20260925 15:17:54.254410 32127 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55294:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:17:54.254675 32127 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55294:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:17:54.258486 32127 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 13c3304caf114421a656a1d9d7bc6024: Sending DeleteTablet for 3 replicas of tablet 7dc4693a133c47b58477e5c1a9b6456a
I20260925 15:17:54.259694 32248 tablet_service.cc:1514] Processing DeleteTablet for tablet 7dc4693a133c47b58477e5c1a9b6456a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:17:54 UTC) from {username='slave'} at 127.0.0.1:49168
I20260925 15:17:54.259723 32509 tablet_service.cc:1514] Processing DeleteTablet for tablet 7dc4693a133c47b58477e5c1a9b6456a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:17:54 UTC) from {username='slave'} at 127.0.0.1:59756
I20260925 15:17:54.259925 32379 tablet_service.cc:1514] Processing DeleteTablet for tablet 7dc4693a133c47b58477e5c1a9b6456a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:17:54 UTC) from {username='slave'} at 127.0.0.1:43242
I20260925 15:17:54.262084 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 32186
I20260925 15:17:54.289309 32616 tablet_replica.cc:330] T 7dc4693a133c47b58477e5c1a9b6456a P faad434d972f4ce394cce46d5462b99c: stopping tablet replica
I20260925 15:17:54.291689 32616 raft_consensus.cc:2229] T 7dc4693a133c47b58477e5c1a9b6456a P faad434d972f4ce394cce46d5462b99c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:17:54.295816 32616 raft_consensus.cc:2258] T 7dc4693a133c47b58477e5c1a9b6456a P faad434d972f4ce394cce46d5462b99c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:17:54.298916 32614 tablet_replica.cc:330] T 7dc4693a133c47b58477e5c1a9b6456a P 50719f6cc39f45f5a35545528fa16a05: stopping tablet replica
I20260925 15:17:54.299515 32614 raft_consensus.cc:2229] T 7dc4693a133c47b58477e5c1a9b6456a P 50719f6cc39f45f5a35545528fa16a05 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:17:54.300148 32614 raft_consensus.cc:2258] T 7dc4693a133c47b58477e5c1a9b6456a P 50719f6cc39f45f5a35545528fa16a05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:17:54.302867 32615 tablet_replica.cc:330] T 7dc4693a133c47b58477e5c1a9b6456a P 24821bc126ca4afdb1d008d349d445de: stopping tablet replica
I20260925 15:17:54.303329 32615 raft_consensus.cc:2229] T 7dc4693a133c47b58477e5c1a9b6456a P 24821bc126ca4afdb1d008d349d445de [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:17:54.303442 32616 ts_tablet_manager.cc:1905] T 7dc4693a133c47b58477e5c1a9b6456a P faad434d972f4ce394cce46d5462b99c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:17:54.303997 32615 raft_consensus.cc:2258] T 7dc4693a133c47b58477e5c1a9b6456a P 24821bc126ca4afdb1d008d349d445de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:17:54.305896 32614 ts_tablet_manager.cc:1905] T 7dc4693a133c47b58477e5c1a9b6456a P 50719f6cc39f45f5a35545528fa16a05: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:17:54.306766 32589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:54.263982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49174 (local address 127.12.8.1:35491)
0925 15:17:54.264213 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:54.264223 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:54.288092 (+ 23869us) server_negotiation.cc:407] Connection header received
0925 15:17:54.289696 (+ 1604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:54.289708 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:54.289868 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:54.289972 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:17:54.292100 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:54.293889 (+ 1789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:54.302390 (+ 8501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:54.302998 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:54.303166 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:54.304871 (+ 1705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:17:54.304884 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:17:54.304905 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:17:54.305702 (+ 797us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:17:54.305769 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:17:54.305799 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:17:54.305927 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:54.306221 (+ 294us) server_negotiation.cc:299] Negotiation successful
0925 15:17:54.306355 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 15:17:54.307428 32615 ts_tablet_manager.cc:1905] T 7dc4693a133c47b58477e5c1a9b6456a P 24821bc126ca4afdb1d008d349d445de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:17:54.307874 32308 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:17:54.315402 32615 ts_tablet_manager.cc:1918] T 7dc4693a133c47b58477e5c1a9b6456a P 24821bc126ca4afdb1d008d349d445de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:17:54.315642 32615 log.cc:1196] T 7dc4693a133c47b58477e5c1a9b6456a P 24821bc126ca4afdb1d008d349d445de: Deleting WAL directory at /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349162370577-12320-0/minicluster-data/ts-2/wal/wals/7dc4693a133c47b58477e5c1a9b6456a
I20260925 15:17:54.316227 32616 ts_tablet_manager.cc:1918] T 7dc4693a133c47b58477e5c1a9b6456a P faad434d972f4ce394cce46d5462b99c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:17:54.316295 32615 ts_tablet_manager.cc:1939] T 7dc4693a133c47b58477e5c1a9b6456a P 24821bc126ca4afdb1d008d349d445de: Deleting consensus metadata
I20260925 15:17:54.316519 32616 log.cc:1196] T 7dc4693a133c47b58477e5c1a9b6456a P faad434d972f4ce394cce46d5462b99c: Deleting WAL directory at /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349162370577-12320-0/minicluster-data/ts-1/wal/wals/7dc4693a133c47b58477e5c1a9b6456a
I20260925 15:17:54.317219 32616 ts_tablet_manager.cc:1939] T 7dc4693a133c47b58477e5c1a9b6456a P faad434d972f4ce394cce46d5462b99c: Deleting consensus metadata
I20260925 15:17:54.318178 32111 catalog_manager.cc:4841] TS 24821bc126ca4afdb1d008d349d445de (127.12.8.3:45339): tablet 7dc4693a133c47b58477e5c1a9b6456a (table test-table [id=8907cb03e3b44c4eaa82b10f852ce7c9]) successfully deleted
I20260925 15:17:54.319347 32111 catalog_manager.cc:4841] TS faad434d972f4ce394cce46d5462b99c (127.12.8.2:39009): tablet 7dc4693a133c47b58477e5c1a9b6456a (table test-table [id=8907cb03e3b44c4eaa82b10f852ce7c9]) successfully deleted
I20260925 15:17:54.401954 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 32186
W20260925 15:17:54.419018 32112 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.8.1:35491 (error 108)
I20260925 15:17:54.419816 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 32317
I20260925 15:17:54.431548 32591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:54.431116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.1:35491 (local address 127.0.0.1:49176)
0925 15:17:54.431292 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:54.431416 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.1:35491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260925 15:17:54.431898 32112 catalog_manager.cc:4584] TS 50719f6cc39f45f5a35545528fa16a05 (127.12.8.1:35491): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7dc4693a133c47b58477e5c1a9b6456a: Network error: Client connection negotiation failed: client connection to 127.12.8.1:35491: connect: Connection refused (error 111)
I20260925 15:17:54.436357 32590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:54.421795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43246 (local address 127.12.8.2:39009)
0925 15:17:54.422050 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:54.422058 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:54.422081 (+ 23us) server_negotiation.cc:407] Connection header received
0925 15:17:54.422160 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:54.422168 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:54.422343 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:54.422468 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:17:54.423318 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:54.424773 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:54.427577 (+ 2804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:54.428472 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:54.428622 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:54.434802 (+ 6180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:17:54.434810 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:17:54.434824 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:17:54.435420 (+ 596us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:17:54.435466 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:17:54.435490 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:17:54.435716 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:54.435864 (+ 148us) server_negotiation.cc:299] Negotiation successful
0925 15:17:54.435982 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260925 15:17:54.437165 32439 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:17:54.485657 32591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:54.485124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.1:35491 (local address 127.0.0.1:49186)
0925 15:17:54.485364 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:54.485476 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.1:35491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260925 15:17:54.486821 32591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:54.486428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.1:35491 (local address 127.0.0.1:49190)
0925 15:17:54.486599 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:54.486683 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.1:35491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260925 15:17:54.526996 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 32317
I20260925 15:17:54.547510 32591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:54.546921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.1:35491 (local address 127.0.0.1:49204)
0925 15:17:54.547177 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:54.547290 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.1:35491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260925 15:17:54.547799 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 32447
I20260925 15:17:54.549396 32591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:54.549051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.1:35491 (local address 127.0.0.1:49210)
0925 15:17:54.549215 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:54.549289 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.1:35491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260925 15:17:54.555902 32586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:54.548490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59778 (local address 127.12.8.3:45339)
0925 15:17:54.548841 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:54.548847 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:54.548868 (+ 21us) server_negotiation.cc:407] Connection header received
0925 15:17:54.548937 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:54.548943 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:54.549055 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:54.549178 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:17:54.549887 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:54.550719 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:54.552645 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:54.553183 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:54.553282 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:54.554463 (+ 1181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:17:54.554471 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:17:54.554485 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:17:54.555064 (+ 579us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:17:54.555132 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:17:54.555151 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:17:54.555230 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:54.555413 (+ 183us) server_negotiation.cc:299] Negotiation successful
0925 15:17:54.555484 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209}
I20260925 15:17:54.556599 32569 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:17:54.630271 32591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:54.629692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.1:35491 (local address 127.0.0.1:49218)
0925 15:17:54.629931 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:54.630026 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.1:35491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260925 15:17:54.631342 32591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:54.630972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.1:35491 (local address 127.0.0.1:49228)
0925 15:17:54.631163 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:54.631242 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.1:35491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260925 15:17:54.676520 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 32447
I20260925 15:17:54.695950 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 32096
I20260925 15:17:54.697109 32157 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:17:54.805344 32591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:54.804716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.1:35491 (local address 127.0.0.1:49236)
0925 15:17:54.804911 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0925 15:17:54.805063 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.1:35491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260925 15:17:54.805414 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 32096
2026-09-25T15:17:54Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2890 ms)
[----------] 6 tests from EncryptionPolicyTest (21395 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349162370577-12320-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:17:54 dist-test-slave-p468 krb5kdc[32625](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:17:54 dist-test-slave-p468 krb5kdc[32625](info): set up 2 sockets
Sep 25 15:17:54 dist-test-slave-p468 krb5kdc[32625](info): commencing operation
krb5kdc: starting...
W20260925 15:17:55.930830 12320 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349162370577-12320-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349162370577-12320-0/krb5kdc/test-user.keytab.
Sep 25 15:17:56 dist-test-slave-p468 krb5kdc[32625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349476, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:17:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:17:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349162370577-12320-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349162370577-12320-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:17:56.178013 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349162370577-12320-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349162370577-12320-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349162370577-12320-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39485
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349162370577-12320-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39485
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:17:56.503985 32638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:17:56.504271 32638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:17:56.504364 32638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:17:56.511358 32638 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:17:56.511457 32638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:17:56.511502 32638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:17:56.511561 32638 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:17:56.511608 32638 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:17:56.521565 32638 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349162370577-12320-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349162370577-12320-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39485
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349162370577-12320-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39485
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349162370577-12320-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:17:56.522761 32638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:17:56.523983 32638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:17:56.535427 32644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:17:56.535506 32643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:17:56.535472 32646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:17:56.536716 32638 server_base.cc:1034] running on GCE node
I20260925 15:17:56.537410 32638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:17:56.538776 32638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:17:56.539976 32638 hybrid_clock.cc:648] HybridClock initialized: now 1790349476539937 us; error 29 us; skew 500 ppm
I20260925 15:17:56.540279 32638 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:17:56 dist-test-slave-p468 krb5kdc[32625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349476, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:17:56.543305 32638 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:17:56.544744 32638 webserver.cc:460] Webserver started at http://127.0.0.1:35171/ using document root <none> and password file <none>
I20260925 15:17:56.545406 32638 fs_manager.cc:359] Metadata directory not provided
I20260925 15:17:56.545552 32638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:17:56.545851 32638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:17:56.548772 32638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349162370577-12320-0/minicluster-data/master-0/data/instance:
uuid: "8e6eeef04e4e4fcf896f5f02c471beea"
format_stamp: "Formatted at 2026-09-25 15:17:56 on dist-test-slave-p468"
server_key: "bee500032212b98d2fb437ed5c6a6e12"
server_key_iv: "a43b5978514c3cb5e485f82a1548a11d"
server_key_version: "encryptionkey@0"
I20260925 15:17:56.549685 32638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349162370577-12320-0/minicluster-data/master-0/wal/instance:
uuid: "8e6eeef04e4e4fcf896f5f02c471beea"
format_stamp: "Formatted at 2026-09-25 15:17:56 on dist-test-slave-p468"
server_key: "bee500032212b98d2fb437ed5c6a6e12"
server_key_iv: "a43b5978514c3cb5e485f82a1548a11d"
server_key_version: "encryptionkey@0"
I20260925 15:17:56.554708 32638 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260925 15:17:56.557967 32653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:17:56.559338 32638 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:17:56.559532 32638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349162370577-12320-0/minicluster-data/master-0/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349162370577-12320-0/minicluster-data/master-0/wal
uuid: "8e6eeef04e4e4fcf896f5f02c471beea"
format_stamp: "Formatted at 2026-09-25 15:17:56 on dist-test-slave-p468"
server_key: "bee500032212b98d2fb437ed5c6a6e12"
server_key_iv: "a43b5978514c3cb5e485f82a1548a11d"
server_key_version: "encryptionkey@0"
I20260925 15:17:56.559721 32638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349162370577-12320-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349162370577-12320-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349162370577-12320-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:17:56.579252 32638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:17:56.583135 32638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:17:56.583423 32638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:17:56.601374 32638 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39485
I20260925 15:17:56.601420 32704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39485 every 8 connection(s)
I20260925 15:17:56.602768 32638 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
I20260925 15:17:56.606700 32705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:17:56.610693 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 32638
I20260925 15:17:56.610945 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349162370577-12320-0/minicluster-data/master-0/wal/instance
I20260925 15:17:56.611735 12320 external_mini_cluster.cc:1428] Setting key 94cf2a29083893a7059e1dc776404438
I20260925 15:17:56.618268 32705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e6eeef04e4e4fcf896f5f02c471beea: Bootstrap starting.
I20260925 15:17:56.621907 32705 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e6eeef04e4e4fcf896f5f02c471beea: Neither blocks nor log segments found. Creating new log.
I20260925 15:17:56.623179 32705 log.cc:824] T 00000000000000000000000000000000 P 8e6eeef04e4e4fcf896f5f02c471beea: Log is configured to *not* fsync() on all Append() calls
I20260925 15:17:56.626679 32705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e6eeef04e4e4fcf896f5f02c471beea: No bootstrap required, opened a new log
Sep 25 15:17:56 dist-test-slave-p468 krb5kdc[32625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349476, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 15:17:56.634537 32705 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e6eeef04e4e4fcf896f5f02c471beea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e6eeef04e4e4fcf896f5f02c471beea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39485 } }
I20260925 15:17:56.634907 32705 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8e6eeef04e4e4fcf896f5f02c471beea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:17:56.635037 32705 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e6eeef04e4e4fcf896f5f02c471beea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e6eeef04e4e4fcf896f5f02c471beea, State: Initialized, Role: FOLLOWER
I20260925 15:17:56.635787 32705 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e6eeef04e4e4fcf896f5f02c471beea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e6eeef04e4e4fcf896f5f02c471beea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39485 } }
I20260925 15:17:56.636076 32705 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8e6eeef04e4e4fcf896f5f02c471beea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:17:56.636222 32705 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e6eeef04e4e4fcf896f5f02c471beea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:17:56.636415 32705 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e6eeef04e4e4fcf896f5f02c471beea [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:17:56.636701 32708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:56.613749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45555 (local address 127.0.0.1:39485)
0925 15:17:56.614617 (+ 868us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:56.614628 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:56.614668 (+ 40us) server_negotiation.cc:407] Connection header received
0925 15:17:56.617113 (+ 2445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:56.617135 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:56.617511 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:56.617894 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:17:56.619727 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:56.621102 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:56.623041 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:56.623923 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:56.624129 (+ 206us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:56.628946 (+ 4817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:17:56.628954 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:17:56.628968 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:17:56.629014 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:17:56.632219 (+ 3205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:17:56.633061 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:17:56.633071 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:17:56.633077 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:17:56.633237 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:17:56.633738 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:17:56.633741 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:17:56.633743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:17:56.634335 (+ 592us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:17:56.634661 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:56.635174 (+ 513us) server_negotiation.cc:299] Negotiation successful
0925 15:17:56.635484 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":648,"thread_start_us":438,"threads_started":1}
I20260925 15:17:56.638875 32705 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e6eeef04e4e4fcf896f5f02c471beea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e6eeef04e4e4fcf896f5f02c471beea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39485 } }
I20260925 15:17:56.639453 32705 leader_election.cc:302] T 00000000000000000000000000000000 P 8e6eeef04e4e4fcf896f5f02c471beea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e6eeef04e4e4fcf896f5f02c471beea; no voters:
I20260925 15:17:56.640174 32705 leader_election.cc:288] T 00000000000000000000000000000000 P 8e6eeef04e4e4fcf896f5f02c471beea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:17:56.640285 32710 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e6eeef04e4e4fcf896f5f02c471beea [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:17:56.640821 12320 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:17:56.642499 32710 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8e6eeef04e4e4fcf896f5f02c471beea [term 1 LEADER]: Becoming Leader. State: Replica: 8e6eeef04e4e4fcf896f5f02c471beea, State: Running, Role: LEADER
I20260925 15:17:56.642673 32705 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e6eeef04e4e4fcf896f5f02c471beea [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:17:56.643114 32710 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e6eeef04e4e4fcf896f5f02c471beea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e6eeef04e4e4fcf896f5f02c471beea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39485 } }
I20260925 15:17:56.651459 32712 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e6eeef04e4e4fcf896f5f02c471beea [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e6eeef04e4e4fcf896f5f02c471beea. Latest consensus state: current_term: 1 leader_uuid: "8e6eeef04e4e4fcf896f5f02c471beea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e6eeef04e4e4fcf896f5f02c471beea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39485 } } }
I20260925 15:17:56.651914 32712 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e6eeef04e4e4fcf896f5f02c471beea [sys.catalog]: This master's current role is: LEADER
I20260925 15:17:56.655118 32711 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e6eeef04e4e4fcf896f5f02c471beea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8e6eeef04e4e4fcf896f5f02c471beea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e6eeef04e4e4fcf896f5f02c471beea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39485 } } }
I20260925 15:17:56.655462 32711 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e6eeef04e4e4fcf896f5f02c471beea [sys.catalog]: This master's current role is: LEADER
I20260925 15:17:56.655664 32722 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:17:56.665879 32722 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:17:56.665969 32708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:56.646046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41749 (local address 127.0.0.1:39485)
0925 15:17:56.646243 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:56.646249 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:56.646901 (+ 652us) server_negotiation.cc:407] Connection header received
0925 15:17:56.647089 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:56.647094 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:56.647168 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:56.647267 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:17:56.648675 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:56.649554 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:56.653955 (+ 4401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:56.654818 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:56.654924 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:56.657870 (+ 2946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:17:56.657875 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:17:56.657878 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:17:56.657945 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:17:56.661579 (+ 3634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:17:56.662474 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:17:56.662482 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:17:56.662486 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:17:56.662626 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:17:56.663259 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:17:56.663264 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:17:56.663268 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:17:56.663776 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:17:56.664066 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:56.665364 (+ 1298us) server_negotiation.cc:299] Negotiation successful
0925 15:17:56.665559 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 15:17:56.678349 32722 catalog_manager.cc:1347] Generated new cluster ID: cb51c950e3e74bcdb1ae2762ca6e66a0
I20260925 15:17:56.678490 32722 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:17:56.696987 32722 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:17:56.698365 32722 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:17:56.706509 32722 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8e6eeef04e4e4fcf896f5f02c471beea: Generated new TSK 0
I20260925 15:17:56.707342 32722 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:17:56.718276 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 32638
I20260925 15:17:56.720665 32700 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:17:56.812626 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 32638
W20260925 15:17:56.829696 32623 connection.cc:535] client connection to 127.0.0.1:39485 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39485): Connection reset by peer (error 104)
2026-09-25T15:17:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1998 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349162370577-12320-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:17:56 dist-test-slave-p468 krb5kdc[32738](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:17:56 dist-test-slave-p468 krb5kdc[32738](info): set up 2 sockets
Sep 25 15:17:56 dist-test-slave-p468 krb5kdc[32738](info): commencing operation
krb5kdc: starting...
W20260925 15:17:57.922595 12320 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349162370577-12320-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349162370577-12320-0/krb5kdc/test-user.keytab.
Sep 25 15:17:58 dist-test-slave-p468 krb5kdc[32738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349478, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:17:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:17:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349162370577-12320-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349162370577-12320-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:17:58.163657 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349162370577-12320-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349162370577-12320-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349162370577-12320-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45529
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349162370577-12320-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45529
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:17:58.480656 32751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:17:58.480922 32751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:17:58.481014 32751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:17:58.488102 32751 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:17:58.488214 32751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:17:58.488269 32751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:17:58.488322 32751 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:17:58.488368 32751 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:17:58.497848 32751 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349162370577-12320-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349162370577-12320-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45529
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349162370577-12320-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45529
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349162370577-12320-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:17:58.499061 32751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:17:58.500260 32751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:17:58.511979 32759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:17:58.512048 32756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:17:58.512068 32757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:17:58.512823 32751 server_base.cc:1034] running on GCE node
I20260925 15:17:58.513527 32751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:17:58.514954 32751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:17:58.516156 32751 hybrid_clock.cc:648] HybridClock initialized: now 1790349478516093 us; error 52 us; skew 500 ppm
I20260925 15:17:58.516494 32751 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:17:58 dist-test-slave-p468 krb5kdc[32738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349478, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:17:58.519572 32751 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:17:58.520941 32751 webserver.cc:460] Webserver started at http://127.0.0.1:35929/ using document root <none> and password file <none>
I20260925 15:17:58.521579 32751 fs_manager.cc:359] Metadata directory not provided
I20260925 15:17:58.521725 32751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:17:58.522027 32751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:17:58.524915 32751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349162370577-12320-0/minicluster-data/master-0/data/instance:
uuid: "c525ce39120d4d82b2cfb2288441caaa"
format_stamp: "Formatted at 2026-09-25 15:17:58 on dist-test-slave-p468"
server_key: "006f46c9a3691bd1eb8db84b0f04c105"
server_key_iv: "3fe2dc127cad169617d5f255c9e98cb5"
server_key_version: "encryptionkey@0"
I20260925 15:17:58.525666 32751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349162370577-12320-0/minicluster-data/master-0/wal/instance:
uuid: "c525ce39120d4d82b2cfb2288441caaa"
format_stamp: "Formatted at 2026-09-25 15:17:58 on dist-test-slave-p468"
server_key: "006f46c9a3691bd1eb8db84b0f04c105"
server_key_iv: "3fe2dc127cad169617d5f255c9e98cb5"
server_key_version: "encryptionkey@0"
I20260925 15:17:58.530606 32751 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260925 15:17:58.533958 32766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:17:58.535233 32751 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:17:58.535429 32751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349162370577-12320-0/minicluster-data/master-0/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349162370577-12320-0/minicluster-data/master-0/wal
uuid: "c525ce39120d4d82b2cfb2288441caaa"
format_stamp: "Formatted at 2026-09-25 15:17:58 on dist-test-slave-p468"
server_key: "006f46c9a3691bd1eb8db84b0f04c105"
server_key_iv: "3fe2dc127cad169617d5f255c9e98cb5"
server_key_version: "encryptionkey@0"
I20260925 15:17:58.535624 32751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349162370577-12320-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349162370577-12320-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349162370577-12320-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:17:58.550686 32751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:17:58.554973 32751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:17:58.555249 32751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:17:58.573254 32751 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45529
I20260925 15:17:58.573297 349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45529 every 8 connection(s)
I20260925 15:17:58.574838 32751 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
I20260925 15:17:58.575151 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 32751
I20260925 15:17:58.575332 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349162370577-12320-0/minicluster-data/master-0/wal/instance
I20260925 15:17:58.575816 12320 external_mini_cluster.cc:1428] Setting key 2a456ce3894331fbc1a79261252eeb2f
I20260925 15:17:58.579303 350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:17:58.588454 353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:58.577665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57915 (local address 127.0.0.1:45529)
0925 15:17:58.578483 (+ 818us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:58.578495 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:58.578535 (+ 40us) server_negotiation.cc:407] Connection header received
0925 15:17:58.580180 (+ 1645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:58.580201 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:58.580563 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:58.580940 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:17:58.581465 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:58.582805 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:58.584147 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:58.584787 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:58.584982 (+ 195us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:58.586303 (+ 1321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:17:58.586311 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:17:58.586323 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:17:58.586855 (+ 532us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:17:58.586928 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:17:58.586954 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:17:58.587129 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:58.587418 (+ 289us) server_negotiation.cc:299] Negotiation successful
0925 15:17:58.587571 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":350,"threads_started":1}
I20260925 15:17:58.589955 350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c525ce39120d4d82b2cfb2288441caaa: Bootstrap starting.
I20260925 15:17:58.592294 12320 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:17:58.593221 350 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c525ce39120d4d82b2cfb2288441caaa: Neither blocks nor log segments found. Creating new log.
I20260925 15:17:58.594174 350 log.cc:824] T 00000000000000000000000000000000 P c525ce39120d4d82b2cfb2288441caaa: Log is configured to *not* fsync() on all Append() calls
I20260925 15:17:58.597445 350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c525ce39120d4d82b2cfb2288441caaa: No bootstrap required, opened a new log
I20260925 15:17:58.602768 353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:17:58.596641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51139 (local address 127.0.0.1:45529)
0925 15:17:58.596822 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 15:17:58.596829 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:17:58.597187 (+ 358us) server_negotiation.cc:407] Connection header received
0925 15:17:58.597400 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:17:58.597404 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:17:58.597461 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:17:58.597561 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:17:58.598014 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:58.598859 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:58.600246 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:17:58.600712 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:17:58.600841 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:17:58.601999 (+ 1158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:17:58.602003 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:17:58.602006 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:17:58.602023 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:17:58.602066 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:17:58.602080 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:17:58.602233 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 15:17:58.602454 (+ 221us) server_negotiation.cc:299] Negotiation successful
0925 15:17:58.602509 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 15:17:58.605085 350 raft_consensus.cc:348] T 00000000000000000000000000000000 P c525ce39120d4d82b2cfb2288441caaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c525ce39120d4d82b2cfb2288441caaa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45529 } }
I20260925 15:17:58.605443 350 raft_consensus.cc:374] T 00000000000000000000000000000000 P c525ce39120d4d82b2cfb2288441caaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:17:58.605548 350 raft_consensus.cc:729] T 00000000000000000000000000000000 P c525ce39120d4d82b2cfb2288441caaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c525ce39120d4d82b2cfb2288441caaa, State: Initialized, Role: FOLLOWER
I20260925 15:17:58.606139 350 consensus_queue.cc:260] T 00000000000000000000000000000000 P c525ce39120d4d82b2cfb2288441caaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c525ce39120d4d82b2cfb2288441caaa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45529 } }
I20260925 15:17:58.606443 350 raft_consensus.cc:388] T 00000000000000000000000000000000 P c525ce39120d4d82b2cfb2288441caaa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:17:58.606581 350 raft_consensus.cc:482] T 00000000000000000000000000000000 P c525ce39120d4d82b2cfb2288441caaa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:17:58.606775 350 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c525ce39120d4d82b2cfb2288441caaa [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:17:58.609189 350 raft_consensus.cc:504] T 00000000000000000000000000000000 P c525ce39120d4d82b2cfb2288441caaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c525ce39120d4d82b2cfb2288441caaa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45529 } }
I20260925 15:17:58.609670 350 leader_election.cc:302] T 00000000000000000000000000000000 P c525ce39120d4d82b2cfb2288441caaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c525ce39120d4d82b2cfb2288441caaa; no voters:
I20260925 15:17:58.610296 350 leader_election.cc:288] T 00000000000000000000000000000000 P c525ce39120d4d82b2cfb2288441caaa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:17:58.610414 360 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c525ce39120d4d82b2cfb2288441caaa [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:17:58.611135 360 raft_consensus.cc:686] T 00000000000000000000000000000000 P c525ce39120d4d82b2cfb2288441caaa [term 1 LEADER]: Becoming Leader. State: Replica: c525ce39120d4d82b2cfb2288441caaa, State: Running, Role: LEADER
I20260925 15:17:58.611636 360 consensus_queue.cc:237] T 00000000000000000000000000000000 P c525ce39120d4d82b2cfb2288441caaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c525ce39120d4d82b2cfb2288441caaa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45529 } }
I20260925 15:17:58.612267 350 sys_catalog.cc:564] T 00000000000000000000000000000000 P c525ce39120d4d82b2cfb2288441caaa [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:17:58.614367 362 sys_catalog.cc:455] T 00000000000000000000000000000000 P c525ce39120d4d82b2cfb2288441caaa [sys.catalog]: SysCatalogTable state changed. Reason: New leader c525ce39120d4d82b2cfb2288441caaa. Latest consensus state: current_term: 1 leader_uuid: "c525ce39120d4d82b2cfb2288441caaa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c525ce39120d4d82b2cfb2288441caaa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45529 } } }
I20260925 15:17:58.614795 362 sys_catalog.cc:458] T 00000000000000000000000000000000 P c525ce39120d4d82b2cfb2288441caaa [sys.catalog]: This master's current role is: LEADER
I20260925 15:17:58.615736 369 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:17:58.615589 361 sys_catalog.cc:455] T 00000000000000000000000000000000 P c525ce39120d4d82b2cfb2288441caaa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c525ce39120d4d82b2cfb2288441caaa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c525ce39120d4d82b2cfb2288441caaa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45529 } } }
I20260925 15:17:58.615993 361 sys_catalog.cc:458] T 00000000000000000000000000000000 P c525ce39120d4d82b2cfb2288441caaa [sys.catalog]: This master's current role is: LEADER
I20260925 15:17:58.621584 369 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:17:58.630155 369 catalog_manager.cc:1347] Generated new cluster ID: d092d8ba72104a4a95b870e9e4963365
I20260925 15:17:58.630298 369 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:17:58.673750 369 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:17:58.675694 369 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:17:58.686262 369 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c525ce39120d4d82b2cfb2288441caaa: Generated new TSK 0
I20260925 15:17:58.687114 369 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:17:58.704310 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 32751
W20260925 15:17:58.705582 345 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:57915
W20260925 15:17:58.706393 12320 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 32751: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 15:17:58.706585 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 32751
W20260925 15:17:58.723428 32736 connection.cc:535] client connection to 127.0.0.1:45529 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45529): Connection reset by peer (error 104)
2026-09-25T15:17:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1893 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349162370577-12320-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:17:58 dist-test-slave-p468 krb5kdc[381](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:17:58 dist-test-slave-p468 krb5kdc[381](info): set up 2 sockets
Sep 25 15:17:58 dist-test-slave-p468 krb5kdc[381](info): commencing operation
krb5kdc: starting...
W20260925 15:17:59.802937 12320 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.007s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349162370577-12320-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349162370577-12320-0/krb5kdc/test-user.keytab.
Sep 25 15:17:59 dist-test-slave-p468 krb5kdc[381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349479, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:17:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:17:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349162370577-12320-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349162370577-12320-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:18:00.056974 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349162370577-12320-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349162370577-12320-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349162370577-12320-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42861
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349162370577-12320-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42861
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:18:00.392753 394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:18:00.393078 394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:18:00.393240 394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:18:00.400437 394 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:18:00.400591 394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:18:00.400699 394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:18:00.400799 394 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:18:00.400885 394 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:18:00.411080 394 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349162370577-12320-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349162370577-12320-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42861
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349162370577-12320-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349162370577-12320-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:18:00.412343 394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:18:00.413641 394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:18:00.425570 399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:00.425531 400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:00.425642 402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:18:00.426448 394 server_base.cc:1034] running on GCE node
I20260925 15:18:00.427209 394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:18:00.428607 394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:18:00.429847 394 hybrid_clock.cc:648] HybridClock initialized: now 1790349480429803 us; error 39 us; skew 500 ppm
I20260925 15:18:00.430241 394 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:18:00 dist-test-slave-p468 krb5kdc[381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349480, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:18:00.437033 394 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:18:00.438587 394 webserver.cc:460] Webserver started at http://127.0.0.1:37317/ using document root <none> and password file <none>
I20260925 15:18:00.439254 394 fs_manager.cc:359] Metadata directory not provided
I20260925 15:18:00.439397 394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:18:00.439697 394 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:18:00.442621 394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349162370577-12320-0/minicluster-data/master-0/data/instance:
uuid: "ba2f4238cc504ca2a24d59352398aa23"
format_stamp: "Formatted at 2026-09-25 15:18:00 on dist-test-slave-p468"
server_key: "fe5b71292502210c006785f290c7ffa6"
server_key_iv: "457aee75df39d63ee49b9d4b646f3915"
server_key_version: "encryptionkey@0"
I20260925 15:18:00.443378 394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349162370577-12320-0/minicluster-data/master-0/wal/instance:
uuid: "ba2f4238cc504ca2a24d59352398aa23"
format_stamp: "Formatted at 2026-09-25 15:18:00 on dist-test-slave-p468"
server_key: "fe5b71292502210c006785f290c7ffa6"
server_key_iv: "457aee75df39d63ee49b9d4b646f3915"
server_key_version: "encryptionkey@0"
I20260925 15:18:00.448242 394 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260925 15:18:00.451583 409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:00.452832 394 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 15:18:00.453027 394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349162370577-12320-0/minicluster-data/master-0/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349162370577-12320-0/minicluster-data/master-0/wal
uuid: "ba2f4238cc504ca2a24d59352398aa23"
format_stamp: "Formatted at 2026-09-25 15:18:00 on dist-test-slave-p468"
server_key: "fe5b71292502210c006785f290c7ffa6"
server_key_iv: "457aee75df39d63ee49b9d4b646f3915"
server_key_version: "encryptionkey@0"
I20260925 15:18:00.453220 394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349162370577-12320-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349162370577-12320-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349162370577-12320-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:18:00.458942 394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:18:00.462941 394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:18:00.463240 394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:18:00.481695 394 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42861
I20260925 15:18:00.481734 460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42861 every 8 connection(s)
I20260925 15:18:00.483141 394 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
I20260925 15:18:00.486832 461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:18:00.489451 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 394
I20260925 15:18:00.489634 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349162370577-12320-0/minicluster-data/master-0/wal/instance
I20260925 15:18:00.490047 12320 external_mini_cluster.cc:1428] Setting key d4715b030f280b262a4dafd8baedd58c
I20260925 15:18:00.497526 464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:00.491793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38559 (local address 127.0.0.1:42861)
0925 15:18:00.492591 (+ 798us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:00.492603 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:00.492642 (+ 39us) server_negotiation.cc:407] Connection header received
0925 15:18:00.494326 (+ 1684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:00.494348 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:00.494675 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:00.494999 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:00.496057 (+ 1058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:00.496063 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:00.496080 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:18:00.496098 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:00.496161 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:18:00.496181 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:00.496353 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:00.496453 (+ 100us) server_negotiation.cc:299] Negotiation successful
0925 15:18:00.496645 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":368,"threads_started":1}
I20260925 15:18:00.498517 461 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba2f4238cc504ca2a24d59352398aa23: Bootstrap starting.
I20260925 15:18:00.501132 12320 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:18:00.501901 461 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba2f4238cc504ca2a24d59352398aa23: Neither blocks nor log segments found. Creating new log.
I20260925 15:18:00.502928 461 log.cc:824] T 00000000000000000000000000000000 P ba2f4238cc504ca2a24d59352398aa23: Log is configured to *not* fsync() on all Append() calls
I20260925 15:18:00.506147 461 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba2f4238cc504ca2a24d59352398aa23: No bootstrap required, opened a new log
I20260925 15:18:00.508345 464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:00.505700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59055 (local address 127.0.0.1:42861)
0925 15:18:00.505852 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:00.505858 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:00.506372 (+ 514us) server_negotiation.cc:407] Connection header received
0925 15:18:00.506575 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:00.506585 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:00.506634 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:00.506746 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:00.507676 (+ 930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:00.507685 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:00.507689 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:18:00.507707 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:00.507759 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:18:00.507774 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:00.507923 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:00.508015 (+ 92us) server_negotiation.cc:299] Negotiation successful
0925 15:18:00.508079 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 15:18:00.513808 461 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba2f4238cc504ca2a24d59352398aa23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba2f4238cc504ca2a24d59352398aa23" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42861 } }
I20260925 15:18:00.514262 461 raft_consensus.cc:374] T 00000000000000000000000000000000 P ba2f4238cc504ca2a24d59352398aa23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:18:00.514375 461 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba2f4238cc504ca2a24d59352398aa23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba2f4238cc504ca2a24d59352398aa23, State: Initialized, Role: FOLLOWER
I20260925 15:18:00.515084 461 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba2f4238cc504ca2a24d59352398aa23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba2f4238cc504ca2a24d59352398aa23" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42861 } }
I20260925 15:18:00.515336 461 raft_consensus.cc:388] T 00000000000000000000000000000000 P ba2f4238cc504ca2a24d59352398aa23 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:18:00.515476 461 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba2f4238cc504ca2a24d59352398aa23 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:18:00.515651 461 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba2f4238cc504ca2a24d59352398aa23 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:18:00.517707 461 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba2f4238cc504ca2a24d59352398aa23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba2f4238cc504ca2a24d59352398aa23" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42861 } }
I20260925 15:18:00.518245 461 leader_election.cc:302] T 00000000000000000000000000000000 P ba2f4238cc504ca2a24d59352398aa23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba2f4238cc504ca2a24d59352398aa23; no voters:
I20260925 15:18:00.518812 461 leader_election.cc:288] T 00000000000000000000000000000000 P ba2f4238cc504ca2a24d59352398aa23 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:18:00.518944 471 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba2f4238cc504ca2a24d59352398aa23 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:18:00.519585 471 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba2f4238cc504ca2a24d59352398aa23 [term 1 LEADER]: Becoming Leader. State: Replica: ba2f4238cc504ca2a24d59352398aa23, State: Running, Role: LEADER
I20260925 15:18:00.520114 471 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba2f4238cc504ca2a24d59352398aa23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba2f4238cc504ca2a24d59352398aa23" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42861 } }
I20260925 15:18:00.521016 461 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba2f4238cc504ca2a24d59352398aa23 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:18:00.523432 473 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba2f4238cc504ca2a24d59352398aa23 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba2f4238cc504ca2a24d59352398aa23. Latest consensus state: current_term: 1 leader_uuid: "ba2f4238cc504ca2a24d59352398aa23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba2f4238cc504ca2a24d59352398aa23" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42861 } } }
I20260925 15:18:00.523404 472 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba2f4238cc504ca2a24d59352398aa23 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ba2f4238cc504ca2a24d59352398aa23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba2f4238cc504ca2a24d59352398aa23" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42861 } } }
I20260925 15:18:00.523882 473 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba2f4238cc504ca2a24d59352398aa23 [sys.catalog]: This master's current role is: LEADER
I20260925 15:18:00.523911 472 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba2f4238cc504ca2a24d59352398aa23 [sys.catalog]: This master's current role is: LEADER
I20260925 15:18:00.525851 478 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:18:00.531229 478 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:18:00.539989 478 catalog_manager.cc:1347] Generated new cluster ID: 412ce495c8c14f0b9a7e586f50f804d6
I20260925 15:18:00.540112 478 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:18:00.557953 478 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:18:00.558503 478 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:18:00.570261 478 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ba2f4238cc504ca2a24d59352398aa23: Generated new TSK 0
I20260925 15:18:00.571278 478 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:18:00.578404 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 394
W20260925 15:18:00.579541 456 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:38559
W20260925 15:18:00.580248 12320 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 394: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 15:18:00.580415 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 394
2026-09-25T15:18:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1873 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349162370577-12320-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:18:00 dist-test-slave-p468 krb5kdc[492](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:18:00 dist-test-slave-p468 krb5kdc[492](info): set up 2 sockets
Sep 25 15:18:00 dist-test-slave-p468 krb5kdc[492](info): commencing operation
krb5kdc: starting...
W20260925 15:18:01.690555 12320 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349162370577-12320-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349162370577-12320-0/krb5kdc/test-user.keytab.
Sep 25 15:18:01 dist-test-slave-p468 krb5kdc[492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349481, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:18:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:18:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349162370577-12320-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349162370577-12320-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:18:01.950412 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349162370577-12320-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349162370577-12320-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349162370577-12320-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42183
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349162370577-12320-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42183
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:18:02.274780 505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:18:02.275089 505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:18:02.275243 505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:18:02.282529 505 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:18:02.282639 505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:18:02.282718 505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:18:02.282801 505 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:18:02.282869 505 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:18:02.292976 505 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349162370577-12320-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349162370577-12320-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42183
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349162370577-12320-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42183
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349162370577-12320-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:18:02.294332 505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:18:02.295615 505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:18:02.307559 510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:02.307585 511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:02.307703 513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:18:02.308328 505 server_base.cc:1034] running on GCE node
I20260925 15:18:02.309116 505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:18:02.310585 505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:18:02.311790 505 hybrid_clock.cc:648] HybridClock initialized: now 1790349482311751 us; error 32 us; skew 500 ppm
I20260925 15:18:02.312145 505 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:18:02 dist-test-slave-p468 krb5kdc[492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349482, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:18:02.315469 505 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:18:02.316942 505 webserver.cc:460] Webserver started at http://127.0.0.1:40497/ using document root <none> and password file <none>
I20260925 15:18:02.317611 505 fs_manager.cc:359] Metadata directory not provided
I20260925 15:18:02.317756 505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:18:02.318063 505 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:18:02.321408 505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349162370577-12320-0/minicluster-data/master-0/data/instance:
uuid: "de15f70a2a4d4cf982a08df6c1725b65"
format_stamp: "Formatted at 2026-09-25 15:18:02 on dist-test-slave-p468"
server_key: "a037ee2b7f6b6c52da1691208229991a"
server_key_iv: "2adca9f2ca9eddd0b845d54506dcee58"
server_key_version: "encryptionkey@0"
I20260925 15:18:02.322168 505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349162370577-12320-0/minicluster-data/master-0/wal/instance:
uuid: "de15f70a2a4d4cf982a08df6c1725b65"
format_stamp: "Formatted at 2026-09-25 15:18:02 on dist-test-slave-p468"
server_key: "a037ee2b7f6b6c52da1691208229991a"
server_key_iv: "2adca9f2ca9eddd0b845d54506dcee58"
server_key_version: "encryptionkey@0"
I20260925 15:18:02.327239 505 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260925 15:18:02.330637 520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:02.331987 505 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260925 15:18:02.332176 505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349162370577-12320-0/minicluster-data/master-0/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349162370577-12320-0/minicluster-data/master-0/wal
uuid: "de15f70a2a4d4cf982a08df6c1725b65"
format_stamp: "Formatted at 2026-09-25 15:18:02 on dist-test-slave-p468"
server_key: "a037ee2b7f6b6c52da1691208229991a"
server_key_iv: "2adca9f2ca9eddd0b845d54506dcee58"
server_key_version: "encryptionkey@0"
I20260925 15:18:02.332376 505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349162370577-12320-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349162370577-12320-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349162370577-12320-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:18:02.353670 505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:18:02.358160 505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:18:02.358493 505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:18:02.376801 505 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42183
I20260925 15:18:02.376839 571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42183 every 8 connection(s)
I20260925 15:18:02.378204 505 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
I20260925 15:18:02.381883 572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:18:02.384871 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 505
I20260925 15:18:02.385100 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349162370577-12320-0/minicluster-data/master-0/wal/instance
I20260925 15:18:02.385653 12320 external_mini_cluster.cc:1428] Setting key 8a1dc40155414678f03cbb0aa803b330
I20260925 15:18:02.393668 572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de15f70a2a4d4cf982a08df6c1725b65: Bootstrap starting.
I20260925 15:18:02.396754 572 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de15f70a2a4d4cf982a08df6c1725b65: Neither blocks nor log segments found. Creating new log.
I20260925 15:18:02.397748 572 log.cc:824] T 00000000000000000000000000000000 P de15f70a2a4d4cf982a08df6c1725b65: Log is configured to *not* fsync() on all Append() calls
Sep 25 15:18:02 dist-test-slave-p468 krb5kdc[492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349481, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 15:18:02.400682 572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de15f70a2a4d4cf982a08df6c1725b65: No bootstrap required, opened a new log
I20260925 15:18:02.407827 575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:02.387594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40207 (local address 127.0.0.1:42183)
0925 15:18:02.388367 (+ 773us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:02.388379 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:02.388420 (+ 41us) server_negotiation.cc:407] Connection header received
0925 15:18:02.390064 (+ 1644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:02.390085 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:02.390541 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:02.390945 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:02.392367 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:02.393662 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:02.395062 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:02.395725 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:02.395949 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:02.400368 (+ 4419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:02.400376 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:02.400390 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:02.400429 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:02.403648 (+ 3219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:02.404402 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:02.404409 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:02.404415 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:02.404586 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:02.405017 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:02.405028 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:02.405031 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:02.405545 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:02.405890 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:02.406357 (+ 467us) server_negotiation.cc:299] Negotiation successful
0925 15:18:02.406794 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":336,"threads_started":1}
I20260925 15:18:02.408650 572 raft_consensus.cc:348] T 00000000000000000000000000000000 P de15f70a2a4d4cf982a08df6c1725b65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de15f70a2a4d4cf982a08df6c1725b65" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42183 } }
I20260925 15:18:02.409019 572 raft_consensus.cc:374] T 00000000000000000000000000000000 P de15f70a2a4d4cf982a08df6c1725b65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:18:02.409169 572 raft_consensus.cc:729] T 00000000000000000000000000000000 P de15f70a2a4d4cf982a08df6c1725b65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de15f70a2a4d4cf982a08df6c1725b65, State: Initialized, Role: FOLLOWER
I20260925 15:18:02.409781 572 consensus_queue.cc:260] T 00000000000000000000000000000000 P de15f70a2a4d4cf982a08df6c1725b65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de15f70a2a4d4cf982a08df6c1725b65" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42183 } }
I20260925 15:18:02.410048 572 raft_consensus.cc:388] T 00000000000000000000000000000000 P de15f70a2a4d4cf982a08df6c1725b65 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:18:02.410245 572 raft_consensus.cc:482] T 00000000000000000000000000000000 P de15f70a2a4d4cf982a08df6c1725b65 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:18:02.410457 572 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de15f70a2a4d4cf982a08df6c1725b65 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:18:02.411621 12320 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:18:02.412937 572 raft_consensus.cc:504] T 00000000000000000000000000000000 P de15f70a2a4d4cf982a08df6c1725b65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de15f70a2a4d4cf982a08df6c1725b65" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42183 } }
I20260925 15:18:02.413426 572 leader_election.cc:302] T 00000000000000000000000000000000 P de15f70a2a4d4cf982a08df6c1725b65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de15f70a2a4d4cf982a08df6c1725b65; no voters:
I20260925 15:18:02.414091 572 leader_election.cc:288] T 00000000000000000000000000000000 P de15f70a2a4d4cf982a08df6c1725b65 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:18:02.414206 577 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de15f70a2a4d4cf982a08df6c1725b65 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:18:02.414885 577 raft_consensus.cc:686] T 00000000000000000000000000000000 P de15f70a2a4d4cf982a08df6c1725b65 [term 1 LEADER]: Becoming Leader. State: Replica: de15f70a2a4d4cf982a08df6c1725b65, State: Running, Role: LEADER
I20260925 15:18:02.415701 577 consensus_queue.cc:237] T 00000000000000000000000000000000 P de15f70a2a4d4cf982a08df6c1725b65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de15f70a2a4d4cf982a08df6c1725b65" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42183 } }
I20260925 15:18:02.418013 577 sys_catalog.cc:455] T 00000000000000000000000000000000 P de15f70a2a4d4cf982a08df6c1725b65 [sys.catalog]: SysCatalogTable state changed. Reason: New leader de15f70a2a4d4cf982a08df6c1725b65. Latest consensus state: current_term: 1 leader_uuid: "de15f70a2a4d4cf982a08df6c1725b65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de15f70a2a4d4cf982a08df6c1725b65" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42183 } } }
I20260925 15:18:02.419896 577 sys_catalog.cc:458] T 00000000000000000000000000000000 P de15f70a2a4d4cf982a08df6c1725b65 [sys.catalog]: This master's current role is: LEADER
I20260925 15:18:02.419456 581 sys_catalog.cc:455] T 00000000000000000000000000000000 P de15f70a2a4d4cf982a08df6c1725b65 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "de15f70a2a4d4cf982a08df6c1725b65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de15f70a2a4d4cf982a08df6c1725b65" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42183 } } }
I20260925 15:18:02.420326 581 sys_catalog.cc:458] T 00000000000000000000000000000000 P de15f70a2a4d4cf982a08df6c1725b65 [sys.catalog]: This master's current role is: LEADER
I20260925 15:18:02.422564 572 sys_catalog.cc:564] T 00000000000000000000000000000000 P de15f70a2a4d4cf982a08df6c1725b65 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:18:02.422865 584 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:18:02.429279 584 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:18:02.440532 575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:02.422440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55269 (local address 127.0.0.1:42183)
0925 15:18:02.422618 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:02.422626 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:02.423029 (+ 403us) server_negotiation.cc:407] Connection header received
0925 15:18:02.423242 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:02.423247 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:02.423324 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:02.423439 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:02.425124 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:02.426332 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:02.428211 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:02.428893 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:02.429006 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:02.432367 (+ 3361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:02.432376 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:02.432385 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:02.432424 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:02.436278 (+ 3854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:02.437328 (+ 1050us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:02.437334 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:02.437348 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:02.437514 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:02.437971 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:02.437978 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:02.437981 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:02.438407 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:02.438644 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:02.439923 (+ 1279us) server_negotiation.cc:299] Negotiation successful
0925 15:18:02.440139 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 15:18:02.443857 584 catalog_manager.cc:1347] Generated new cluster ID: 98e3ffd123174a7fa27c315d8147cd8b
I20260925 15:18:02.444043 584 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:18:02.454023 584 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:18:02.455904 584 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:18:02.467334 584 catalog_manager.cc:5868] T 00000000000000000000000000000000 P de15f70a2a4d4cf982a08df6c1725b65: Generated new TSK 0
I20260925 15:18:02.468163 584 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:18:02.477403 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 505
I20260925 15:18:02.478771 567 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:18:02.567951 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 505
2026-09-25T15:18:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1988 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349162370577-12320-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:18:02 dist-test-slave-p468 krb5kdc[604](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:18:02 dist-test-slave-p468 krb5kdc[604](info): set up 2 sockets
Sep 25 15:18:02 dist-test-slave-p468 krb5kdc[604](info): commencing operation
krb5kdc: starting...
W20260925 15:18:03.670568 12320 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349162370577-12320-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349162370577-12320-0/krb5kdc/test-user.keytab.
Sep 25 15:18:03 dist-test-slave-p468 krb5kdc[604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349483, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:18:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:18:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349162370577-12320-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349162370577-12320-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:18:03.917163 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349162370577-12320-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349162370577-12320-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349162370577-12320-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44499
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349162370577-12320-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44499
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:18:04.240340 617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:18:04.240672 617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:18:04.240841 617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:18:04.248226 617 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:18:04.248381 617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:18:04.248486 617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:18:04.248595 617 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:18:04.248682 617 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:18:04.258352 617 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349162370577-12320-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349162370577-12320-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44499
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349162370577-12320-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44499
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349162370577-12320-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:18:04.259639 617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:18:04.260960 617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:18:04.272780 623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:04.272925 625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:04.272904 622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:18:04.273526 617 server_base.cc:1034] running on GCE node
I20260925 15:18:04.274353 617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:18:04.275734 617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:18:04.276950 617 hybrid_clock.cc:648] HybridClock initialized: now 1790349484276911 us; error 31 us; skew 500 ppm
I20260925 15:18:04.277261 617 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:18:04 dist-test-slave-p468 krb5kdc[604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349484, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:18:04.280295 617 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:18:04.281780 617 webserver.cc:460] Webserver started at http://127.0.0.1:40963/ using document root <none> and password file <none>
I20260925 15:18:04.282483 617 fs_manager.cc:359] Metadata directory not provided
I20260925 15:18:04.282586 617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:18:04.282840 617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:18:04.285728 617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349162370577-12320-0/minicluster-data/master-0/data/instance:
uuid: "287e0a1f1c5d45c89288f561faa146e6"
format_stamp: "Formatted at 2026-09-25 15:18:04 on dist-test-slave-p468"
server_key: "01d458b709dd1603310924b531deb9eb"
server_key_iv: "baff390437c9ca0d50f5c40674814257"
server_key_version: "encryptionkey@0"
I20260925 15:18:04.286525 617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349162370577-12320-0/minicluster-data/master-0/wal/instance:
uuid: "287e0a1f1c5d45c89288f561faa146e6"
format_stamp: "Formatted at 2026-09-25 15:18:04 on dist-test-slave-p468"
server_key: "01d458b709dd1603310924b531deb9eb"
server_key_iv: "baff390437c9ca0d50f5c40674814257"
server_key_version: "encryptionkey@0"
I20260925 15:18:04.291507 617 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260925 15:18:04.294835 632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:04.296106 617 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 15:18:04.296262 617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349162370577-12320-0/minicluster-data/master-0/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349162370577-12320-0/minicluster-data/master-0/wal
uuid: "287e0a1f1c5d45c89288f561faa146e6"
format_stamp: "Formatted at 2026-09-25 15:18:04 on dist-test-slave-p468"
server_key: "01d458b709dd1603310924b531deb9eb"
server_key_iv: "baff390437c9ca0d50f5c40674814257"
server_key_version: "encryptionkey@0"
I20260925 15:18:04.296490 617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349162370577-12320-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349162370577-12320-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349162370577-12320-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:18:04.314078 617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:18:04.318100 617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:18:04.318357 617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:18:04.336632 617 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44499
I20260925 15:18:04.336668 683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44499 every 8 connection(s)
I20260925 15:18:04.338021 617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
I20260925 15:18:04.340016 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 617
I20260925 15:18:04.340209 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349162370577-12320-0/minicluster-data/master-0/wal/instance
I20260925 15:18:04.340786 12320 external_mini_cluster.cc:1428] Setting key 2bfe729d23f73c291b230e9f1bf493c1
I20260925 15:18:04.342769 684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:18:04.354034 687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:04.342714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51761 (local address 127.0.0.1:44499)
0925 15:18:04.343528 (+ 814us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:04.343539 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:04.343576 (+ 37us) server_negotiation.cc:407] Connection header received
0925 15:18:04.345159 (+ 1583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:04.345179 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:04.345652 (+ 473us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:04.345988 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:04.346557 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:04.347847 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:04.349705 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:04.350420 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:04.350548 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:04.351776 (+ 1228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:04.351783 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:04.351795 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:18:04.352332 (+ 537us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:04.352403 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:18:04.352430 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:04.352524 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:04.352754 (+ 230us) server_negotiation.cc:299] Negotiation successful
0925 15:18:04.353065 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":380,"threads_started":1}
I20260925 15:18:04.355917 684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 287e0a1f1c5d45c89288f561faa146e6: Bootstrap starting.
I20260925 15:18:04.357789 12320 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:18:04.359372 684 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 287e0a1f1c5d45c89288f561faa146e6: Neither blocks nor log segments found. Creating new log.
I20260925 15:18:04.360352 684 log.cc:824] T 00000000000000000000000000000000 P 287e0a1f1c5d45c89288f561faa146e6: Log is configured to *not* fsync() on all Append() calls
I20260925 15:18:04.363494 684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 287e0a1f1c5d45c89288f561faa146e6: No bootstrap required, opened a new log
I20260925 15:18:04.368713 687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:04.362066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38927 (local address 127.0.0.1:44499)
0925 15:18:04.362271 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:04.362277 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:04.362793 (+ 516us) server_negotiation.cc:407] Connection header received
0925 15:18:04.362938 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:04.362945 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:04.363029 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:04.363138 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:04.363612 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:04.364709 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:04.365994 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:04.366783 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:04.366914 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:04.367900 (+ 986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:04.367907 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:04.367912 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:18:04.367932 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:04.367984 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:18:04.368006 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:04.368107 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:04.368228 (+ 121us) server_negotiation.cc:299] Negotiation successful
0925 15:18:04.368355 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 15:18:04.372614 684 raft_consensus.cc:348] T 00000000000000000000000000000000 P 287e0a1f1c5d45c89288f561faa146e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "287e0a1f1c5d45c89288f561faa146e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44499 } }
I20260925 15:18:04.372969 684 raft_consensus.cc:374] T 00000000000000000000000000000000 P 287e0a1f1c5d45c89288f561faa146e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:18:04.373049 684 raft_consensus.cc:729] T 00000000000000000000000000000000 P 287e0a1f1c5d45c89288f561faa146e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 287e0a1f1c5d45c89288f561faa146e6, State: Initialized, Role: FOLLOWER
I20260925 15:18:04.373529 684 consensus_queue.cc:260] T 00000000000000000000000000000000 P 287e0a1f1c5d45c89288f561faa146e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "287e0a1f1c5d45c89288f561faa146e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44499 } }
I20260925 15:18:04.373724 684 raft_consensus.cc:388] T 00000000000000000000000000000000 P 287e0a1f1c5d45c89288f561faa146e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:18:04.373834 684 raft_consensus.cc:482] T 00000000000000000000000000000000 P 287e0a1f1c5d45c89288f561faa146e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:18:04.373950 684 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 287e0a1f1c5d45c89288f561faa146e6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:18:04.376093 684 raft_consensus.cc:504] T 00000000000000000000000000000000 P 287e0a1f1c5d45c89288f561faa146e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "287e0a1f1c5d45c89288f561faa146e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44499 } }
I20260925 15:18:04.376518 684 leader_election.cc:302] T 00000000000000000000000000000000 P 287e0a1f1c5d45c89288f561faa146e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 287e0a1f1c5d45c89288f561faa146e6; no voters:
I20260925 15:18:04.377017 684 leader_election.cc:288] T 00000000000000000000000000000000 P 287e0a1f1c5d45c89288f561faa146e6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:18:04.377168 694 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 287e0a1f1c5d45c89288f561faa146e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:18:04.378623 694 raft_consensus.cc:686] T 00000000000000000000000000000000 P 287e0a1f1c5d45c89288f561faa146e6 [term 1 LEADER]: Becoming Leader. State: Replica: 287e0a1f1c5d45c89288f561faa146e6, State: Running, Role: LEADER
I20260925 15:18:04.379215 684 sys_catalog.cc:564] T 00000000000000000000000000000000 P 287e0a1f1c5d45c89288f561faa146e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:18:04.379165 694 consensus_queue.cc:237] T 00000000000000000000000000000000 P 287e0a1f1c5d45c89288f561faa146e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "287e0a1f1c5d45c89288f561faa146e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44499 } }
I20260925 15:18:04.381807 695 sys_catalog.cc:455] T 00000000000000000000000000000000 P 287e0a1f1c5d45c89288f561faa146e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "287e0a1f1c5d45c89288f561faa146e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "287e0a1f1c5d45c89288f561faa146e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44499 } } }
I20260925 15:18:04.381880 696 sys_catalog.cc:455] T 00000000000000000000000000000000 P 287e0a1f1c5d45c89288f561faa146e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 287e0a1f1c5d45c89288f561faa146e6. Latest consensus state: current_term: 1 leader_uuid: "287e0a1f1c5d45c89288f561faa146e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "287e0a1f1c5d45c89288f561faa146e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44499 } } }
I20260925 15:18:04.382382 696 sys_catalog.cc:458] T 00000000000000000000000000000000 P 287e0a1f1c5d45c89288f561faa146e6 [sys.catalog]: This master's current role is: LEADER
I20260925 15:18:04.382385 695 sys_catalog.cc:458] T 00000000000000000000000000000000 P 287e0a1f1c5d45c89288f561faa146e6 [sys.catalog]: This master's current role is: LEADER
I20260925 15:18:04.383301 703 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:18:04.388407 703 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:18:04.396193 703 catalog_manager.cc:1347] Generated new cluster ID: 0e289bb89dcb4a91a53616a14cc303a9
I20260925 15:18:04.396299 703 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:18:04.415553 703 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:18:04.417490 703 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:18:04.426231 703 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 287e0a1f1c5d45c89288f561faa146e6: Generated new TSK 0
I20260925 15:18:04.427376 703 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:18:04.437758 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 617
W20260925 15:18:04.439297 679 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:51761
W20260925 15:18:04.439962 12320 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 617: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 15:18:04.440100 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 617
2026-09-25T15:18:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1871 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349162370577-12320-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:18:04 dist-test-slave-p468 krb5kdc[715](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:18:04 dist-test-slave-p468 krb5kdc[715](info): set up 2 sockets
Sep 25 15:18:04 dist-test-slave-p468 krb5kdc[715](info): commencing operation
krb5kdc: starting...
W20260925 15:18:05.562644 12320 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.001s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349162370577-12320-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349162370577-12320-0/krb5kdc/test-user.keytab.
Sep 25 15:18:05 dist-test-slave-p468 krb5kdc[715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:18:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:18:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349162370577-12320-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349162370577-12320-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:18:05.814522 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349162370577-12320-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349162370577-12320-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349162370577-12320-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39007
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349162370577-12320-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39007
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:18:06.142164 728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:18:06.142443 728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:18:06.142635 728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:18:06.149595 728 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:18:06.149726 728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:18:06.149839 728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:18:06.149964 728 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:18:06.150038 728 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:18:06.159423 728 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349162370577-12320-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349162370577-12320-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39007
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349162370577-12320-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349162370577-12320-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:18:06.160661 728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:18:06.161924 728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:18:06.173218 734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:06.173264 736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:06.173334 733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:18:06.173983 728 server_base.cc:1034] running on GCE node
I20260925 15:18:06.174757 728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:18:06.176146 728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:18:06.177361 728 hybrid_clock.cc:648] HybridClock initialized: now 1790349486177293 us; error 55 us; skew 500 ppm
I20260925 15:18:06.177703 728 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:18:06 dist-test-slave-p468 krb5kdc[715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349486, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:18:06.180801 728 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:18:06.182143 728 webserver.cc:460] Webserver started at http://127.0.0.1:35701/ using document root <none> and password file <none>
I20260925 15:18:06.182823 728 fs_manager.cc:359] Metadata directory not provided
I20260925 15:18:06.182961 728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:18:06.183254 728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:18:06.186080 728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349162370577-12320-0/minicluster-data/master-0/data/instance:
uuid: "b4892117ac694ce895af916a57ebc002"
format_stamp: "Formatted at 2026-09-25 15:18:06 on dist-test-slave-p468"
server_key: "96da1d03526407911dcfcd08c189bd69"
server_key_iv: "dbf5c754c1700961a5f85f44a7cc58fa"
server_key_version: "encryptionkey@0"
I20260925 15:18:06.186944 728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349162370577-12320-0/minicluster-data/master-0/wal/instance:
uuid: "b4892117ac694ce895af916a57ebc002"
format_stamp: "Formatted at 2026-09-25 15:18:06 on dist-test-slave-p468"
server_key: "96da1d03526407911dcfcd08c189bd69"
server_key_iv: "dbf5c754c1700961a5f85f44a7cc58fa"
server_key_version: "encryptionkey@0"
I20260925 15:18:06.191848 728 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 15:18:06.195262 743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:06.196573 728 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 15:18:06.196768 728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349162370577-12320-0/minicluster-data/master-0/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349162370577-12320-0/minicluster-data/master-0/wal
uuid: "b4892117ac694ce895af916a57ebc002"
format_stamp: "Formatted at 2026-09-25 15:18:06 on dist-test-slave-p468"
server_key: "96da1d03526407911dcfcd08c189bd69"
server_key_iv: "dbf5c754c1700961a5f85f44a7cc58fa"
server_key_version: "encryptionkey@0"
I20260925 15:18:06.196967 728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349162370577-12320-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349162370577-12320-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349162370577-12320-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:18:06.202317 728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:18:06.205966 728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:18:06.206312 728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:18:06.225569 728 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39007
I20260925 15:18:06.225586 794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39007 every 8 connection(s)
I20260925 15:18:06.227021 728 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
I20260925 15:18:06.227469 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 728
I20260925 15:18:06.227664 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349162370577-12320-0/minicluster-data/master-0/wal/instance
I20260925 15:18:06.228416 12320 external_mini_cluster.cc:1428] Setting key bcf03729784e2dbb37e5e722eba39743
I20260925 15:18:06.231384 795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:18:06.236325 798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:06.230368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38171 (local address 127.0.0.1:39007)
0925 15:18:06.231148 (+ 780us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:06.231159 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:06.231199 (+ 40us) server_negotiation.cc:407] Connection header received
0925 15:18:06.232897 (+ 1698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:06.232920 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:06.233235 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:06.233632 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:06.234712 (+ 1080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:06.234718 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:06.234733 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:18:06.234747 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:06.234831 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:18:06.234854 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:06.235099 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:06.235236 (+ 137us) server_negotiation.cc:299] Negotiation successful
0925 15:18:06.235424 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":352,"threads_started":1}
I20260925 15:18:06.239933 12320 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:18:06.242564 795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4892117ac694ce895af916a57ebc002: Bootstrap starting.
I20260925 15:18:06.246416 795 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b4892117ac694ce895af916a57ebc002: Neither blocks nor log segments found. Creating new log.
I20260925 15:18:06.246770 798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:06.244266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57825 (local address 127.0.0.1:39007)
0925 15:18:06.244489 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:06.244497 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:06.244816 (+ 319us) server_negotiation.cc:407] Connection header received
0925 15:18:06.244997 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:06.245000 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:06.245046 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:06.245118 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:06.246042 (+ 924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:06.246046 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:06.246049 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:18:06.246062 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:06.246100 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:18:06.246114 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:06.246254 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:06.246393 (+ 139us) server_negotiation.cc:299] Negotiation successful
0925 15:18:06.246442 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260925 15:18:06.247613 795 log.cc:824] T 00000000000000000000000000000000 P b4892117ac694ce895af916a57ebc002: Log is configured to *not* fsync() on all Append() calls
I20260925 15:18:06.250363 795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4892117ac694ce895af916a57ebc002: No bootstrap required, opened a new log
I20260925 15:18:06.257346 795 raft_consensus.cc:348] T 00000000000000000000000000000000 P b4892117ac694ce895af916a57ebc002 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4892117ac694ce895af916a57ebc002" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39007 } }
I20260925 15:18:06.257728 795 raft_consensus.cc:374] T 00000000000000000000000000000000 P b4892117ac694ce895af916a57ebc002 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:18:06.257838 795 raft_consensus.cc:729] T 00000000000000000000000000000000 P b4892117ac694ce895af916a57ebc002 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4892117ac694ce895af916a57ebc002, State: Initialized, Role: FOLLOWER
I20260925 15:18:06.258430 795 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4892117ac694ce895af916a57ebc002 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4892117ac694ce895af916a57ebc002" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39007 } }
I20260925 15:18:06.258669 795 raft_consensus.cc:388] T 00000000000000000000000000000000 P b4892117ac694ce895af916a57ebc002 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:18:06.258821 795 raft_consensus.cc:482] T 00000000000000000000000000000000 P b4892117ac694ce895af916a57ebc002 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:18:06.258993 795 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b4892117ac694ce895af916a57ebc002 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:18:06.261134 795 raft_consensus.cc:504] T 00000000000000000000000000000000 P b4892117ac694ce895af916a57ebc002 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4892117ac694ce895af916a57ebc002" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39007 } }
I20260925 15:18:06.261616 795 leader_election.cc:302] T 00000000000000000000000000000000 P b4892117ac694ce895af916a57ebc002 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b4892117ac694ce895af916a57ebc002; no voters:
I20260925 15:18:06.262153 795 leader_election.cc:288] T 00000000000000000000000000000000 P b4892117ac694ce895af916a57ebc002 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:18:06.262333 805 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b4892117ac694ce895af916a57ebc002 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:18:06.263046 805 raft_consensus.cc:686] T 00000000000000000000000000000000 P b4892117ac694ce895af916a57ebc002 [term 1 LEADER]: Becoming Leader. State: Replica: b4892117ac694ce895af916a57ebc002, State: Running, Role: LEADER
I20260925 15:18:06.263609 805 consensus_queue.cc:237] T 00000000000000000000000000000000 P b4892117ac694ce895af916a57ebc002 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4892117ac694ce895af916a57ebc002" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39007 } }
I20260925 15:18:06.264250 795 sys_catalog.cc:564] T 00000000000000000000000000000000 P b4892117ac694ce895af916a57ebc002 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:18:06.266456 807 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4892117ac694ce895af916a57ebc002 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4892117ac694ce895af916a57ebc002. Latest consensus state: current_term: 1 leader_uuid: "b4892117ac694ce895af916a57ebc002" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4892117ac694ce895af916a57ebc002" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39007 } } }
I20260925 15:18:06.266355 806 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4892117ac694ce895af916a57ebc002 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b4892117ac694ce895af916a57ebc002" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4892117ac694ce895af916a57ebc002" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39007 } } }
I20260925 15:18:06.266836 806 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4892117ac694ce895af916a57ebc002 [sys.catalog]: This master's current role is: LEADER
I20260925 15:18:06.266836 807 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4892117ac694ce895af916a57ebc002 [sys.catalog]: This master's current role is: LEADER
I20260925 15:18:06.269158 812 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:18:06.274084 812 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:18:06.283255 812 catalog_manager.cc:1347] Generated new cluster ID: 33a11e1f79b647b980caf28667a00893
I20260925 15:18:06.283412 812 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:18:06.294971 812 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:18:06.295385 812 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:18:06.309755 812 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b4892117ac694ce895af916a57ebc002: Generated new TSK 0
I20260925 15:18:06.310895 812 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:18:06.321604 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 728
W20260925 15:18:06.322813 790 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:38171
W20260925 15:18:06.323550 12320 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 728: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 15:18:06.323702 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 728
2026-09-25T15:18:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1883 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349162370577-12320-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:18:06 dist-test-slave-p468 krb5kdc[826](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:18:06 dist-test-slave-p468 krb5kdc[826](info): set up 2 sockets
Sep 25 15:18:06 dist-test-slave-p468 krb5kdc[826](info): commencing operation
krb5kdc: starting...
W20260925 15:18:07.422644 12320 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.001s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349162370577-12320-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349162370577-12320-0/krb5kdc/test-user.keytab.
Sep 25 15:18:07 dist-test-slave-p468 krb5kdc[826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349487, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:18:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:18:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349162370577-12320-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349162370577-12320-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:18:07.683810 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349162370577-12320-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349162370577-12320-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349162370577-12320-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45215
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349162370577-12320-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45215
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:18:08.010541 839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:18:08.010810 839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:18:08.011013 839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:18:08.018018 839 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:18:08.018121 839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:18:08.018249 839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:18:08.018339 839 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:18:08.018425 839 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:18:08.028038 839 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349162370577-12320-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349162370577-12320-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45215
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349162370577-12320-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349162370577-12320-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:18:08.029304 839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:18:08.030592 839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:18:08.042274 844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:08.042297 845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:08.042238 847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:18:08.043251 839 server_base.cc:1034] running on GCE node
I20260925 15:18:08.044023 839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:18:08.045480 839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:18:08.046701 839 hybrid_clock.cc:648] HybridClock initialized: now 1790349488046661 us; error 56 us; skew 500 ppm
I20260925 15:18:08.047072 839 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:18:08 dist-test-slave-p468 krb5kdc[826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349488, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:18:08.050148 839 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:18:08.051746 839 webserver.cc:460] Webserver started at http://127.0.0.1:42901/ using document root <none> and password file <none>
I20260925 15:18:08.052500 839 fs_manager.cc:359] Metadata directory not provided
I20260925 15:18:08.052655 839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:18:08.052971 839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:18:08.056272 839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349162370577-12320-0/minicluster-data/master-0/data/instance:
uuid: "bda0f9ce20554cab9220cb7a006f9515"
format_stamp: "Formatted at 2026-09-25 15:18:08 on dist-test-slave-p468"
server_key: "90eaf953989a381bc8650a2340dc0e26"
server_key_iv: "1ac29c6339f62307ec5ba4bc0b9fed9e"
server_key_version: "encryptionkey@0"
I20260925 15:18:08.057144 839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349162370577-12320-0/minicluster-data/master-0/wal/instance:
uuid: "bda0f9ce20554cab9220cb7a006f9515"
format_stamp: "Formatted at 2026-09-25 15:18:08 on dist-test-slave-p468"
server_key: "90eaf953989a381bc8650a2340dc0e26"
server_key_iv: "1ac29c6339f62307ec5ba4bc0b9fed9e"
server_key_version: "encryptionkey@0"
I20260925 15:18:08.062304 839 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 15:18:08.065555 854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:08.066844 839 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:18:08.067049 839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349162370577-12320-0/minicluster-data/master-0/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349162370577-12320-0/minicluster-data/master-0/wal
uuid: "bda0f9ce20554cab9220cb7a006f9515"
format_stamp: "Formatted at 2026-09-25 15:18:08 on dist-test-slave-p468"
server_key: "90eaf953989a381bc8650a2340dc0e26"
server_key_iv: "1ac29c6339f62307ec5ba4bc0b9fed9e"
server_key_version: "encryptionkey@0"
I20260925 15:18:08.067235 839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349162370577-12320-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349162370577-12320-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349162370577-12320-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:18:08.080772 839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:18:08.084424 839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:18:08.084699 839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:18:08.102561 839 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45215
I20260925 15:18:08.102617 905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45215 every 8 connection(s)
I20260925 15:18:08.103828 839 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
I20260925 15:18:08.107278 906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:18:08.109901 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 839
I20260925 15:18:08.110086 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349162370577-12320-0/minicluster-data/master-0/wal/instance
I20260925 15:18:08.110654 12320 external_mini_cluster.cc:1428] Setting key bac0d379b2b01231e24f20096af6240c
I20260925 15:18:08.119632 906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bda0f9ce20554cab9220cb7a006f9515: Bootstrap starting.
I20260925 15:18:08.123466 906 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bda0f9ce20554cab9220cb7a006f9515: Neither blocks nor log segments found. Creating new log.
I20260925 15:18:08.124699 906 log.cc:824] T 00000000000000000000000000000000 P bda0f9ce20554cab9220cb7a006f9515: Log is configured to *not* fsync() on all Append() calls
Sep 25 15:18:08 dist-test-slave-p468 krb5kdc[826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349487, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 15:18:08.127568 906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bda0f9ce20554cab9220cb7a006f9515: No bootstrap required, opened a new log
I20260925 15:18:08.133340 909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:08.112298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.242:58851 (local address 127.0.0.1:45215)
0925 15:18:08.113076 (+ 778us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:08.113088 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:08.113129 (+ 41us) server_negotiation.cc:407] Connection header received
0925 15:18:08.114826 (+ 1697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:08.114846 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:08.115209 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:08.115497 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:08.117249 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:08.118631 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:08.120363 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:08.121005 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:08.121148 (+ 143us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:08.126323 (+ 5175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:08.126330 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:08.126343 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:08.126378 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:08.129402 (+ 3024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:08.130211 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:08.130221 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:08.130230 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:08.130405 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:08.130869 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:08.130874 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:08.130877 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:08.131363 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:08.131706 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:08.132151 (+ 445us) server_negotiation.cc:299] Negotiation successful
0925 15:18:08.132446 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":348,"threads_started":1}
I20260925 15:18:08.135187 906 raft_consensus.cc:348] T 00000000000000000000000000000000 P bda0f9ce20554cab9220cb7a006f9515 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda0f9ce20554cab9220cb7a006f9515" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45215 } }
I20260925 15:18:08.135536 906 raft_consensus.cc:374] T 00000000000000000000000000000000 P bda0f9ce20554cab9220cb7a006f9515 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:18:08.135660 906 raft_consensus.cc:729] T 00000000000000000000000000000000 P bda0f9ce20554cab9220cb7a006f9515 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bda0f9ce20554cab9220cb7a006f9515, State: Initialized, Role: FOLLOWER
I20260925 15:18:08.136253 906 consensus_queue.cc:260] T 00000000000000000000000000000000 P bda0f9ce20554cab9220cb7a006f9515 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda0f9ce20554cab9220cb7a006f9515" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45215 } }
I20260925 15:18:08.136549 906 raft_consensus.cc:388] T 00000000000000000000000000000000 P bda0f9ce20554cab9220cb7a006f9515 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:18:08.136700 906 raft_consensus.cc:482] T 00000000000000000000000000000000 P bda0f9ce20554cab9220cb7a006f9515 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:18:08.136895 906 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bda0f9ce20554cab9220cb7a006f9515 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:18:08.137086 12320 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:18:08.139099 906 raft_consensus.cc:504] T 00000000000000000000000000000000 P bda0f9ce20554cab9220cb7a006f9515 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda0f9ce20554cab9220cb7a006f9515" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45215 } }
I20260925 15:18:08.139638 906 leader_election.cc:302] T 00000000000000000000000000000000 P bda0f9ce20554cab9220cb7a006f9515 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bda0f9ce20554cab9220cb7a006f9515; no voters:
I20260925 15:18:08.140287 906 leader_election.cc:288] T 00000000000000000000000000000000 P bda0f9ce20554cab9220cb7a006f9515 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:18:08.140383 914 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bda0f9ce20554cab9220cb7a006f9515 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:18:08.142787 906 sys_catalog.cc:564] T 00000000000000000000000000000000 P bda0f9ce20554cab9220cb7a006f9515 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:18:08.147321 914 raft_consensus.cc:686] T 00000000000000000000000000000000 P bda0f9ce20554cab9220cb7a006f9515 [term 1 LEADER]: Becoming Leader. State: Replica: bda0f9ce20554cab9220cb7a006f9515, State: Running, Role: LEADER
I20260925 15:18:08.147996 914 consensus_queue.cc:237] T 00000000000000000000000000000000 P bda0f9ce20554cab9220cb7a006f9515 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda0f9ce20554cab9220cb7a006f9515" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45215 } }
I20260925 15:18:08.151597 916 sys_catalog.cc:455] T 00000000000000000000000000000000 P bda0f9ce20554cab9220cb7a006f9515 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bda0f9ce20554cab9220cb7a006f9515" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda0f9ce20554cab9220cb7a006f9515" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45215 } } }
I20260925 15:18:08.152118 916 sys_catalog.cc:458] T 00000000000000000000000000000000 P bda0f9ce20554cab9220cb7a006f9515 [sys.catalog]: This master's current role is: LEADER
I20260925 15:18:08.157198 923 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:18:08.157817 909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:08.141539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.242:55735 (local address 127.0.0.1:45215)
0925 15:18:08.141751 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:08.141759 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:08.142296 (+ 537us) server_negotiation.cc:407] Connection header received
0925 15:18:08.142510 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:08.142523 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:08.142598 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:08.142680 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:08.144240 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:08.144986 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:08.146227 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:08.146703 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:08.146818 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:08.149290 (+ 2472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:08.149295 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:08.149299 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:08.149325 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:08.153321 (+ 3996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:08.154606 (+ 1285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:08.154616 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:08.154623 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:08.154986 (+ 363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:08.155726 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:08.155733 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:08.155739 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:08.156206 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:08.157305 (+ 1099us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:08.157453 (+ 148us) server_negotiation.cc:299] Negotiation successful
0925 15:18:08.157581 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 15:18:08.158246 917 sys_catalog.cc:455] T 00000000000000000000000000000000 P bda0f9ce20554cab9220cb7a006f9515 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bda0f9ce20554cab9220cb7a006f9515. Latest consensus state: current_term: 1 leader_uuid: "bda0f9ce20554cab9220cb7a006f9515" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda0f9ce20554cab9220cb7a006f9515" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45215 } } }
I20260925 15:18:08.158627 917 sys_catalog.cc:458] T 00000000000000000000000000000000 P bda0f9ce20554cab9220cb7a006f9515 [sys.catalog]: This master's current role is: LEADER
I20260925 15:18:08.162725 923 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:18:08.171926 923 catalog_manager.cc:1347] Generated new cluster ID: 42813800e3194940aeac5c494a86c6f2
I20260925 15:18:08.172051 923 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:18:08.202428 923 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:18:08.203826 923 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:18:08.213658 923 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bda0f9ce20554cab9220cb7a006f9515: Generated new TSK 0
I20260925 15:18:08.214695 923 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:18:08.227984 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 839
I20260925 15:18:08.229110 901 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:18:08.321714 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 839
W20260925 15:18:08.338892 824 connection.cc:535] client connection to 127.0.0.1:45215 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45215): Connection reset by peer (error 104)
2026-09-25T15:18:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1998 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349162370577-12320-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:18:08 dist-test-slave-p468 krb5kdc[938](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:18:08 dist-test-slave-p468 krb5kdc[938](info): set up 2 sockets
Sep 25 15:18:08 dist-test-slave-p468 krb5kdc[938](info): commencing operation
krb5kdc: starting...
W20260925 15:18:09.419013 12320 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349162370577-12320-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349162370577-12320-0/krb5kdc/test-user.keytab.
Sep 25 15:18:09 dist-test-slave-p468 krb5kdc[938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:18:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:18:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349162370577-12320-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349162370577-12320-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:18:09.669548 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349162370577-12320-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349162370577-12320-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349162370577-12320-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44539
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349162370577-12320-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44539
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:18:10.003799 951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:18:10.004089 951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:18:10.004264 951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:18:10.011987 951 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:18:10.012094 951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:18:10.012204 951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:18:10.012318 951 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:18:10.012387 951 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:18:10.021991 951 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349162370577-12320-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349162370577-12320-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44539
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349162370577-12320-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349162370577-12320-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:18:10.023298 951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:18:10.024622 951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:18:10.036931 956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:10.036906 957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:10.036850 959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:18:10.037926 951 server_base.cc:1034] running on GCE node
I20260925 15:18:10.038760 951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:18:10.040105 951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:18:10.041316 951 hybrid_clock.cc:648] HybridClock initialized: now 1790349490041279 us; error 32 us; skew 500 ppm
I20260925 15:18:10.041690 951 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:18:10 dist-test-slave-p468 krb5kdc[938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349490, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:18:10.044909 951 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:18:10.046576 951 webserver.cc:460] Webserver started at http://127.0.0.1:32831/ using document root <none> and password file <none>
I20260925 15:18:10.047272 951 fs_manager.cc:359] Metadata directory not provided
I20260925 15:18:10.047420 951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:18:10.047735 951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:18:10.050817 951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349162370577-12320-0/minicluster-data/master-0/data/instance:
uuid: "88e957d7f3c1441f930d8df33f7a4741"
format_stamp: "Formatted at 2026-09-25 15:18:10 on dist-test-slave-p468"
server_key: "46a6cfd91a582c43ad0fe369e7282bb7"
server_key_iv: "dfeebcc634a7f0a56f3a28e7144776cf"
server_key_version: "encryptionkey@0"
I20260925 15:18:10.051599 951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349162370577-12320-0/minicluster-data/master-0/wal/instance:
uuid: "88e957d7f3c1441f930d8df33f7a4741"
format_stamp: "Formatted at 2026-09-25 15:18:10 on dist-test-slave-p468"
server_key: "46a6cfd91a582c43ad0fe369e7282bb7"
server_key_iv: "dfeebcc634a7f0a56f3a28e7144776cf"
server_key_version: "encryptionkey@0"
I20260925 15:18:10.056730 951 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 15:18:10.060351 966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:10.061648 951 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 15:18:10.061834 951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349162370577-12320-0/minicluster-data/master-0/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349162370577-12320-0/minicluster-data/master-0/wal
uuid: "88e957d7f3c1441f930d8df33f7a4741"
format_stamp: "Formatted at 2026-09-25 15:18:10 on dist-test-slave-p468"
server_key: "46a6cfd91a582c43ad0fe369e7282bb7"
server_key_iv: "dfeebcc634a7f0a56f3a28e7144776cf"
server_key_version: "encryptionkey@0"
I20260925 15:18:10.062031 951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349162370577-12320-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349162370577-12320-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349162370577-12320-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:18:10.091748 951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:18:10.096019 951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:18:10.096314 951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:18:10.115198 951 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44539
I20260925 15:18:10.115236 1017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44539 every 8 connection(s)
I20260925 15:18:10.116554 951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
I20260925 15:18:10.120345 1018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:18:10.123570 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 951
I20260925 15:18:10.123759 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349162370577-12320-0/minicluster-data/master-0/wal/instance
I20260925 15:18:10.124290 12320 external_mini_cluster.cc:1428] Setting key 6c8ce5f3307206698725c943cd02019d
I20260925 15:18:10.131713 1018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88e957d7f3c1441f930d8df33f7a4741: Bootstrap starting.
I20260925 15:18:10.134737 1018 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88e957d7f3c1441f930d8df33f7a4741: Neither blocks nor log segments found. Creating new log.
I20260925 15:18:10.135744 1018 log.cc:824] T 00000000000000000000000000000000 P 88e957d7f3c1441f930d8df33f7a4741: Log is configured to *not* fsync() on all Append() calls
I20260925 15:18:10.136952 1021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:10.126121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.242:52067 (local address 127.0.0.1:44539)
0925 15:18:10.126826 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:10.126837 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:10.126875 (+ 38us) server_negotiation.cc:407] Connection header received
0925 15:18:10.128489 (+ 1614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:10.128508 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:10.128897 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:10.129286 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:10.129831 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:10.131152 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:10.132556 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:10.133230 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:10.133446 (+ 216us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:10.134696 (+ 1250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:10.134703 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:10.134716 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:18:10.135305 (+ 589us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:10.135370 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:18:10.135393 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:10.135571 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:10.135846 (+ 275us) server_negotiation.cc:299] Negotiation successful
0925 15:18:10.136014 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":291,"threads_started":1}
I20260925 15:18:10.138514 1018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88e957d7f3c1441f930d8df33f7a4741: No bootstrap required, opened a new log
I20260925 15:18:10.140630 12320 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:18:10.147717 1018 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88e957d7f3c1441f930d8df33f7a4741 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e957d7f3c1441f930d8df33f7a4741" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44539 } }
I20260925 15:18:10.148229 1018 raft_consensus.cc:374] T 00000000000000000000000000000000 P 88e957d7f3c1441f930d8df33f7a4741 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:18:10.148351 1018 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88e957d7f3c1441f930d8df33f7a4741 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88e957d7f3c1441f930d8df33f7a4741, State: Initialized, Role: FOLLOWER
I20260925 15:18:10.149031 1018 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88e957d7f3c1441f930d8df33f7a4741 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e957d7f3c1441f930d8df33f7a4741" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44539 } }
I20260925 15:18:10.149349 1018 raft_consensus.cc:388] T 00000000000000000000000000000000 P 88e957d7f3c1441f930d8df33f7a4741 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:18:10.149518 1018 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88e957d7f3c1441f930d8df33f7a4741 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:18:10.149721 1018 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88e957d7f3c1441f930d8df33f7a4741 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:18:10.151881 1021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:10.144939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.242:57157 (local address 127.0.0.1:44539)
0925 15:18:10.145159 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:10.145167 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:10.145728 (+ 561us) server_negotiation.cc:407] Connection header received
0925 15:18:10.145942 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:10.145946 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:10.146002 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:10.146113 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:10.146598 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:10.147404 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:10.149099 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:10.149758 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:10.149897 (+ 139us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:10.151106 (+ 1209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:10.151114 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:10.151126 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:18:10.151144 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:10.151177 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:18:10.151191 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:10.151321 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:10.151505 (+ 184us) server_negotiation.cc:299] Negotiation successful
0925 15:18:10.151564 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 15:18:10.152233 1018 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88e957d7f3c1441f930d8df33f7a4741 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e957d7f3c1441f930d8df33f7a4741" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44539 } }
I20260925 15:18:10.152731 1018 leader_election.cc:302] T 00000000000000000000000000000000 P 88e957d7f3c1441f930d8df33f7a4741 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 88e957d7f3c1441f930d8df33f7a4741; no voters:
I20260925 15:18:10.153342 1018 leader_election.cc:288] T 00000000000000000000000000000000 P 88e957d7f3c1441f930d8df33f7a4741 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:18:10.153436 1028 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 88e957d7f3c1441f930d8df33f7a4741 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:18:10.154217 1028 raft_consensus.cc:686] T 00000000000000000000000000000000 P 88e957d7f3c1441f930d8df33f7a4741 [term 1 LEADER]: Becoming Leader. State: Replica: 88e957d7f3c1441f930d8df33f7a4741, State: Running, Role: LEADER
I20260925 15:18:10.154800 1028 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88e957d7f3c1441f930d8df33f7a4741 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e957d7f3c1441f930d8df33f7a4741" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44539 } }
I20260925 15:18:10.155762 1018 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88e957d7f3c1441f930d8df33f7a4741 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:18:10.157869 1029 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88e957d7f3c1441f930d8df33f7a4741 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "88e957d7f3c1441f930d8df33f7a4741" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e957d7f3c1441f930d8df33f7a4741" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44539 } } }
I20260925 15:18:10.158300 1029 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88e957d7f3c1441f930d8df33f7a4741 [sys.catalog]: This master's current role is: LEADER
I20260925 15:18:10.159113 1030 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88e957d7f3c1441f930d8df33f7a4741 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e957d7f3c1441f930d8df33f7a4741. Latest consensus state: current_term: 1 leader_uuid: "88e957d7f3c1441f930d8df33f7a4741" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e957d7f3c1441f930d8df33f7a4741" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44539 } } }
I20260925 15:18:10.159514 1030 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88e957d7f3c1441f930d8df33f7a4741 [sys.catalog]: This master's current role is: LEADER
I20260925 15:18:10.160332 1033 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:18:10.164918 1033 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:18:10.173615 1033 catalog_manager.cc:1347] Generated new cluster ID: 348fb3a70ef14e769c0fb8bbb4c349c2
I20260925 15:18:10.173728 1033 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:18:10.183231 1033 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:18:10.184435 1033 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:18:10.201978 1033 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 88e957d7f3c1441f930d8df33f7a4741: Generated new TSK 0
I20260925 15:18:10.202876 1033 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:18:10.215550 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 951
W20260925 15:18:10.216655 1013 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.242:52067
W20260925 15:18:10.217429 12320 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 951: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 15:18:10.217586 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 951
W20260925 15:18:10.232583 936 connection.cc:535] client connection to 127.0.0.1:44539 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44539): Connection reset by peer (error 104)
2026-09-25T15:18:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1893 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349162370577-12320-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:18:10 dist-test-slave-p468 krb5kdc[1049](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:18:10 dist-test-slave-p468 krb5kdc[1049](info): set up 2 sockets
Sep 25 15:18:10 dist-test-slave-p468 krb5kdc[1049](info): commencing operation
krb5kdc: starting...
W20260925 15:18:11.326733 12320 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.005s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349162370577-12320-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349162370577-12320-0/krb5kdc/test-user.keytab.
Sep 25 15:18:11 dist-test-slave-p468 krb5kdc[1049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349491, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:18:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:18:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349162370577-12320-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349162370577-12320-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:18:11.577307 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349162370577-12320-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349162370577-12320-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349162370577-12320-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39271
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349162370577-12320-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39271
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:18:11.898046 1062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:18:11.898404 1062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:18:11.898563 1062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:18:11.905730 1062 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:18:11.905879 1062 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:18:11.905997 1062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:18:11.906112 1062 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:18:11.906198 1062 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:18:11.915858 1062 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349162370577-12320-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349162370577-12320-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39271
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349162370577-12320-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349162370577-12320-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:18:11.917110 1062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:18:11.918919 1062 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:18:11.930028 1068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:11.930125 1067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:11.930071 1070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:18:11.931064 1062 server_base.cc:1034] running on GCE node
I20260925 15:18:11.931783 1062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:18:11.933071 1062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:18:11.934304 1062 hybrid_clock.cc:648] HybridClock initialized: now 1790349491934270 us; error 51 us; skew 500 ppm
I20260925 15:18:11.934654 1062 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:18:11 dist-test-slave-p468 krb5kdc[1049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349491, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:18:11.942920 1062 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:18:11.944386 1062 webserver.cc:460] Webserver started at http://127.0.0.1:43849/ using document root <none> and password file <none>
I20260925 15:18:11.945072 1062 fs_manager.cc:359] Metadata directory not provided
I20260925 15:18:11.945220 1062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:18:11.945523 1062 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:18:11.948592 1062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349162370577-12320-0/minicluster-data/master-0/data/instance:
uuid: "ff10e94011854e46b93ad35e5f8b46ec"
format_stamp: "Formatted at 2026-09-25 15:18:11 on dist-test-slave-p468"
server_key: "a55bc21d0e1bfcf60e34398116cc20ba"
server_key_iv: "1cf3b4390663723516cdbb94cd8bdf00"
server_key_version: "encryptionkey@0"
I20260925 15:18:11.949357 1062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349162370577-12320-0/minicluster-data/master-0/wal/instance:
uuid: "ff10e94011854e46b93ad35e5f8b46ec"
format_stamp: "Formatted at 2026-09-25 15:18:11 on dist-test-slave-p468"
server_key: "a55bc21d0e1bfcf60e34398116cc20ba"
server_key_iv: "1cf3b4390663723516cdbb94cd8bdf00"
server_key_version: "encryptionkey@0"
I20260925 15:18:11.954420 1062 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 15:18:11.957661 1077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:11.959147 1062 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:18:11.959326 1062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349162370577-12320-0/minicluster-data/master-0/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349162370577-12320-0/minicluster-data/master-0/wal
uuid: "ff10e94011854e46b93ad35e5f8b46ec"
format_stamp: "Formatted at 2026-09-25 15:18:11 on dist-test-slave-p468"
server_key: "a55bc21d0e1bfcf60e34398116cc20ba"
server_key_iv: "1cf3b4390663723516cdbb94cd8bdf00"
server_key_version: "encryptionkey@0"
I20260925 15:18:11.959517 1062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349162370577-12320-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349162370577-12320-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349162370577-12320-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:18:11.965119 1062 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:18:11.968860 1062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:18:11.969151 1062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:18:11.987350 1062 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39271
I20260925 15:18:11.987428 1128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39271 every 8 connection(s)
I20260925 15:18:11.988680 1062 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
I20260925 15:18:11.992383 1129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:18:11.998772 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 1062
I20260925 15:18:11.998972 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349162370577-12320-0/minicluster-data/master-0/wal/instance
I20260925 15:18:11.999534 12320 external_mini_cluster.cc:1428] Setting key 8f71e8372431d6dc241e13ab3ce60a90
I20260925 15:18:12.003422 1129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff10e94011854e46b93ad35e5f8b46ec: Bootstrap starting.
I20260925 15:18:12.006971 1129 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff10e94011854e46b93ad35e5f8b46ec: Neither blocks nor log segments found. Creating new log.
I20260925 15:18:12.007232 1132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:12.001330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.242:32965 (local address 127.0.0.1:39271)
0925 15:18:12.002134 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:12.002145 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:12.002232 (+ 87us) server_negotiation.cc:407] Connection header received
0925 15:18:12.003930 (+ 1698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:12.003953 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:12.004275 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:12.004585 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:12.005661 (+ 1076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:12.005667 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:12.005680 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:18:12.005693 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:12.005748 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:18:12.005767 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:12.005931 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:12.006052 (+ 121us) server_negotiation.cc:299] Negotiation successful
0925 15:18:12.006271 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":364,"threads_started":1}
I20260925 15:18:12.008117 1129 log.cc:824] T 00000000000000000000000000000000 P ff10e94011854e46b93ad35e5f8b46ec: Log is configured to *not* fsync() on all Append() calls
I20260925 15:18:12.011003 12320 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:18:12.011112 1129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff10e94011854e46b93ad35e5f8b46ec: No bootstrap required, opened a new log
I20260925 15:18:12.018662 1132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:12.016293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.242:40181 (local address 127.0.0.1:39271)
0925 15:18:12.016443 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:12.016447 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:12.016799 (+ 352us) server_negotiation.cc:407] Connection header received
0925 15:18:12.016952 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:12.016956 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:12.017003 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:12.017084 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:12.017959 (+ 875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:12.017963 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:12.017966 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:18:12.017983 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:12.018019 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:18:12.018036 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:12.018137 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:12.018295 (+ 158us) server_negotiation.cc:299] Negotiation successful
0925 15:18:12.018339 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 15:18:12.020897 1129 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff10e94011854e46b93ad35e5f8b46ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff10e94011854e46b93ad35e5f8b46ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39271 } }
I20260925 15:18:12.021272 1129 raft_consensus.cc:374] T 00000000000000000000000000000000 P ff10e94011854e46b93ad35e5f8b46ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:18:12.021385 1129 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff10e94011854e46b93ad35e5f8b46ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff10e94011854e46b93ad35e5f8b46ec, State: Initialized, Role: FOLLOWER
I20260925 15:18:12.021945 1129 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff10e94011854e46b93ad35e5f8b46ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff10e94011854e46b93ad35e5f8b46ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39271 } }
I20260925 15:18:12.022178 1129 raft_consensus.cc:388] T 00000000000000000000000000000000 P ff10e94011854e46b93ad35e5f8b46ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:18:12.022359 1129 raft_consensus.cc:482] T 00000000000000000000000000000000 P ff10e94011854e46b93ad35e5f8b46ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:18:12.022527 1129 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff10e94011854e46b93ad35e5f8b46ec [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:18:12.024688 1129 raft_consensus.cc:504] T 00000000000000000000000000000000 P ff10e94011854e46b93ad35e5f8b46ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff10e94011854e46b93ad35e5f8b46ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39271 } }
I20260925 15:18:12.025193 1129 leader_election.cc:302] T 00000000000000000000000000000000 P ff10e94011854e46b93ad35e5f8b46ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ff10e94011854e46b93ad35e5f8b46ec; no voters:
I20260925 15:18:12.025722 1129 leader_election.cc:288] T 00000000000000000000000000000000 P ff10e94011854e46b93ad35e5f8b46ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:18:12.025854 1139 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ff10e94011854e46b93ad35e5f8b46ec [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:18:12.026540 1139 raft_consensus.cc:686] T 00000000000000000000000000000000 P ff10e94011854e46b93ad35e5f8b46ec [term 1 LEADER]: Becoming Leader. State: Replica: ff10e94011854e46b93ad35e5f8b46ec, State: Running, Role: LEADER
I20260925 15:18:12.027073 1139 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff10e94011854e46b93ad35e5f8b46ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff10e94011854e46b93ad35e5f8b46ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39271 } }
I20260925 15:18:12.027755 1129 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff10e94011854e46b93ad35e5f8b46ec [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:18:12.031627 1141 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff10e94011854e46b93ad35e5f8b46ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff10e94011854e46b93ad35e5f8b46ec. Latest consensus state: current_term: 1 leader_uuid: "ff10e94011854e46b93ad35e5f8b46ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff10e94011854e46b93ad35e5f8b46ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39271 } } }
I20260925 15:18:12.031582 1140 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff10e94011854e46b93ad35e5f8b46ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ff10e94011854e46b93ad35e5f8b46ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff10e94011854e46b93ad35e5f8b46ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39271 } } }
I20260925 15:18:12.032105 1140 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff10e94011854e46b93ad35e5f8b46ec [sys.catalog]: This master's current role is: LEADER
I20260925 15:18:12.032114 1141 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff10e94011854e46b93ad35e5f8b46ec [sys.catalog]: This master's current role is: LEADER
I20260925 15:18:12.032917 1149 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:18:12.038345 1149 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:18:12.047327 1149 catalog_manager.cc:1347] Generated new cluster ID: c4f9be28de5c4a42a5dda1ce762781e8
I20260925 15:18:12.047482 1149 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:18:12.071620 1149 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:18:12.072093 1149 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:18:12.086733 1149 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ff10e94011854e46b93ad35e5f8b46ec: Generated new TSK 0
I20260925 15:18:12.087538 1149 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:18:12.098733 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 1062
W20260925 15:18:12.099892 1124 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.242:32965
W20260925 15:18:12.100600 12320 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 1062: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 15:18:12.100760 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 1062
2026-09-25T15:18:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1881 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349162370577-12320-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:18:12 dist-test-slave-p468 krb5kdc[1160](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:18:12 dist-test-slave-p468 krb5kdc[1160](info): set up 2 sockets
Sep 25 15:18:12 dist-test-slave-p468 krb5kdc[1160](info): commencing operation
krb5kdc: starting...
W20260925 15:18:13.194579 12320 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.001s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349162370577-12320-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349162370577-12320-0/krb5kdc/test-user.keytab.
Sep 25 15:18:13 dist-test-slave-p468 krb5kdc[1160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349493, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:18:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:18:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349162370577-12320-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349162370577-12320-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:18:13.449133 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349162370577-12320-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349162370577-12320-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349162370577-12320-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40791
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349162370577-12320-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40791
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:18:13.765622 1173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:18:13.765908 1173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:18:13.766074 1173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:18:13.773250 1173 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:18:13.773348 1173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:18:13.773434 1173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:18:13.773546 1173 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:18:13.773613 1173 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:18:13.783372 1173 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349162370577-12320-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349162370577-12320-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40791
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349162370577-12320-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349162370577-12320-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:18:13.784665 1173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:18:13.785951 1173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:18:13.797729 1178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:13.797912 1179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:13.797974 1181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:18:13.799011 1173 server_base.cc:1034] running on GCE node
I20260925 15:18:13.799818 1173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:18:13.801216 1173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:18:13.802439 1173 hybrid_clock.cc:648] HybridClock initialized: now 1790349493802405 us; error 60 us; skew 500 ppm
I20260925 15:18:13.802785 1173 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:18:13 dist-test-slave-p468 krb5kdc[1160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349493, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:18:13.805838 1173 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:18:13.807358 1173 webserver.cc:460] Webserver started at http://127.0.0.1:42657/ using document root <none> and password file <none>
I20260925 15:18:13.808007 1173 fs_manager.cc:359] Metadata directory not provided
I20260925 15:18:13.808159 1173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:18:13.808449 1173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:18:13.811517 1173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349162370577-12320-0/minicluster-data/master-0/data/instance:
uuid: "b998c14a58c543ff8745f8be41da9618"
format_stamp: "Formatted at 2026-09-25 15:18:13 on dist-test-slave-p468"
server_key: "b7b33603540d1d8686db11b63bbca0b7"
server_key_iv: "51c9d048e25fe03362937d995911f1db"
server_key_version: "encryptionkey@0"
I20260925 15:18:13.812285 1173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349162370577-12320-0/minicluster-data/master-0/wal/instance:
uuid: "b998c14a58c543ff8745f8be41da9618"
format_stamp: "Formatted at 2026-09-25 15:18:13 on dist-test-slave-p468"
server_key: "b7b33603540d1d8686db11b63bbca0b7"
server_key_iv: "51c9d048e25fe03362937d995911f1db"
server_key_version: "encryptionkey@0"
I20260925 15:18:13.817147 1173 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 15:18:13.820415 1188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:13.821666 1173 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 15:18:13.821852 1173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349162370577-12320-0/minicluster-data/master-0/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349162370577-12320-0/minicluster-data/master-0/wal
uuid: "b998c14a58c543ff8745f8be41da9618"
format_stamp: "Formatted at 2026-09-25 15:18:13 on dist-test-slave-p468"
server_key: "b7b33603540d1d8686db11b63bbca0b7"
server_key_iv: "51c9d048e25fe03362937d995911f1db"
server_key_version: "encryptionkey@0"
I20260925 15:18:13.822044 1173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349162370577-12320-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349162370577-12320-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349162370577-12320-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:18:13.845801 1173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:18:13.850165 1173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:18:13.850540 1173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:18:13.871255 1173 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40791
I20260925 15:18:13.871287 1239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40791 every 8 connection(s)
I20260925 15:18:13.872614 1173 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
I20260925 15:18:13.876566 1240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:18:13.881285 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 1173
I20260925 15:18:13.881474 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349162370577-12320-0/minicluster-data/master-0/wal/instance
I20260925 15:18:13.881906 12320 external_mini_cluster.cc:1428] Setting key 9d991c297e2737acacf13b9c11968a9d
I20260925 15:18:13.887028 1240 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b998c14a58c543ff8745f8be41da9618: Bootstrap starting.
I20260925 15:18:13.890592 1240 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b998c14a58c543ff8745f8be41da9618: Neither blocks nor log segments found. Creating new log.
I20260925 15:18:13.891779 1240 log.cc:824] T 00000000000000000000000000000000 P b998c14a58c543ff8745f8be41da9618: Log is configured to *not* fsync() on all Append() calls
I20260925 15:18:13.895298 1240 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b998c14a58c543ff8745f8be41da9618: No bootstrap required, opened a new log
Sep 25 15:18:13 dist-test-slave-p468 krb5kdc[1160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349493, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 15:18:13.903559 1240 raft_consensus.cc:348] T 00000000000000000000000000000000 P b998c14a58c543ff8745f8be41da9618 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b998c14a58c543ff8745f8be41da9618" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40791 } }
I20260925 15:18:13.903909 1240 raft_consensus.cc:374] T 00000000000000000000000000000000 P b998c14a58c543ff8745f8be41da9618 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:18:13.904043 1240 raft_consensus.cc:729] T 00000000000000000000000000000000 P b998c14a58c543ff8745f8be41da9618 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b998c14a58c543ff8745f8be41da9618, State: Initialized, Role: FOLLOWER
I20260925 15:18:13.904718 1240 consensus_queue.cc:260] T 00000000000000000000000000000000 P b998c14a58c543ff8745f8be41da9618 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b998c14a58c543ff8745f8be41da9618" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40791 } }
I20260925 15:18:13.905016 1240 raft_consensus.cc:388] T 00000000000000000000000000000000 P b998c14a58c543ff8745f8be41da9618 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:18:13.905176 1240 raft_consensus.cc:482] T 00000000000000000000000000000000 P b998c14a58c543ff8745f8be41da9618 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:18:13.905380 1240 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b998c14a58c543ff8745f8be41da9618 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:18:13.906112 1243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:13.883858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.242:34537 (local address 127.0.0.1:40791)
0925 15:18:13.884677 (+ 819us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:13.884687 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:13.884733 (+ 46us) server_negotiation.cc:407] Connection header received
0925 15:18:13.886457 (+ 1724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:13.886478 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:13.886870 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:13.887224 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:13.888862 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:13.890111 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:13.892009 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:13.893048 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:13.893208 (+ 160us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:13.898125 (+ 4917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:13.898132 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:13.898146 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:13.898244 (+ 98us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:13.901314 (+ 3070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:13.902090 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:13.902096 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:13.902102 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:13.902347 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:13.902824 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:13.902829 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:13.902833 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:13.903332 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:13.903659 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:13.904284 (+ 625us) server_negotiation.cc:299] Negotiation successful
0925 15:18:13.904625 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":364,"threads_started":1}
I20260925 15:18:13.908383 1240 raft_consensus.cc:504] T 00000000000000000000000000000000 P b998c14a58c543ff8745f8be41da9618 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b998c14a58c543ff8745f8be41da9618" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40791 } }
I20260925 15:18:13.908993 1240 leader_election.cc:302] T 00000000000000000000000000000000 P b998c14a58c543ff8745f8be41da9618 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b998c14a58c543ff8745f8be41da9618; no voters:
I20260925 15:18:13.909703 1240 leader_election.cc:288] T 00000000000000000000000000000000 P b998c14a58c543ff8745f8be41da9618 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:18:13.909890 1245 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b998c14a58c543ff8745f8be41da9618 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:18:13.910466 12320 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:18:13.911454 1245 raft_consensus.cc:686] T 00000000000000000000000000000000 P b998c14a58c543ff8745f8be41da9618 [term 1 LEADER]: Becoming Leader. State: Replica: b998c14a58c543ff8745f8be41da9618, State: Running, Role: LEADER
I20260925 15:18:13.912038 1240 sys_catalog.cc:564] T 00000000000000000000000000000000 P b998c14a58c543ff8745f8be41da9618 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:18:13.911957 1245 consensus_queue.cc:237] T 00000000000000000000000000000000 P b998c14a58c543ff8745f8be41da9618 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b998c14a58c543ff8745f8be41da9618" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40791 } }
I20260925 15:18:13.914528 1246 sys_catalog.cc:455] T 00000000000000000000000000000000 P b998c14a58c543ff8745f8be41da9618 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b998c14a58c543ff8745f8be41da9618" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b998c14a58c543ff8745f8be41da9618" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40791 } } }
I20260925 15:18:13.914896 1246 sys_catalog.cc:458] T 00000000000000000000000000000000 P b998c14a58c543ff8745f8be41da9618 [sys.catalog]: This master's current role is: LEADER
I20260925 15:18:13.915382 1247 sys_catalog.cc:455] T 00000000000000000000000000000000 P b998c14a58c543ff8745f8be41da9618 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b998c14a58c543ff8745f8be41da9618. Latest consensus state: current_term: 1 leader_uuid: "b998c14a58c543ff8745f8be41da9618" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b998c14a58c543ff8745f8be41da9618" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40791 } } }
I20260925 15:18:13.915668 1247 sys_catalog.cc:458] T 00000000000000000000000000000000 P b998c14a58c543ff8745f8be41da9618 [sys.catalog]: This master's current role is: LEADER
I20260925 15:18:13.917407 1251 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:18:13.924458 1251 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:18:13.943128 1251 catalog_manager.cc:1347] Generated new cluster ID: 37a157d47bc64010b070ad63100808b2
I20260925 15:18:13.943362 1251 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:18:13.962896 1243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:13.944358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.242:40317 (local address 127.0.0.1:40791)
0925 15:18:13.944547 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:13.944555 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:13.945158 (+ 603us) server_negotiation.cc:407] Connection header received
0925 15:18:13.945458 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:13.945464 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:13.945547 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:13.945621 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:13.947760 (+ 2139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:13.948949 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:13.950904 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:13.951574 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:13.951683 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:13.955246 (+ 3563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:13.955253 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:13.955258 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:13.955297 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:13.959157 (+ 3860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:13.960149 (+ 992us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:13.960154 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:13.960158 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:13.960326 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:13.960906 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:13.960912 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:13.960915 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:13.961342 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:13.961614 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:13.962316 (+ 702us) server_negotiation.cc:299] Negotiation successful
0925 15:18:13.962518 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 15:18:13.983302 1251 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:18:13.984596 1251 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:18:13.993570 1251 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b998c14a58c543ff8745f8be41da9618: Generated new TSK 0
I20260925 15:18:13.994657 1251 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:18:14.001080 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 1173
I20260925 15:18:14.002301 1235 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:18:14.092939 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 1173
W20260925 15:18:14.109530 1158 connection.cc:535] client connection to 127.0.0.1:40791 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40791): Connection reset by peer (error 104)
2026-09-25T15:18:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1994 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349162370577-12320-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:18:14 dist-test-slave-p468 krb5kdc[1272](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:18:14 dist-test-slave-p468 krb5kdc[1272](info): set up 2 sockets
Sep 25 15:18:14 dist-test-slave-p468 krb5kdc[1272](info): commencing operation
krb5kdc: starting...
W20260925 15:18:15.206552 12320 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349162370577-12320-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349162370577-12320-0/krb5kdc/test-user.keytab.
Sep 25 15:18:15 dist-test-slave-p468 krb5kdc[1272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349495, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:18:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:18:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349162370577-12320-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349162370577-12320-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:18:15.457381 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349162370577-12320-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349162370577-12320-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349162370577-12320-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38813
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349162370577-12320-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38813
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:18:15.793615 1285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:18:15.793908 1285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:18:15.794090 1285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:18:15.801322 1285 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:18:15.801429 1285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:18:15.801535 1285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:18:15.801625 1285 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:18:15.801712 1285 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:18:15.811517 1285 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349162370577-12320-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349162370577-12320-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38813
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349162370577-12320-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349162370577-12320-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:18:15.812757 1285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:18:15.814026 1285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:18:15.825587 1291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:15.825719 1293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:15.825682 1290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:18:15.826517 1285 server_base.cc:1034] running on GCE node
I20260925 15:18:15.827308 1285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:18:15.828619 1285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:18:15.829818 1285 hybrid_clock.cc:648] HybridClock initialized: now 1790349495829776 us; error 39 us; skew 500 ppm
I20260925 15:18:15.830227 1285 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:18:15 dist-test-slave-p468 krb5kdc[1272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349495, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:18:15.833189 1285 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:18:15.834645 1285 webserver.cc:460] Webserver started at http://127.0.0.1:37073/ using document root <none> and password file <none>
I20260925 15:18:15.835245 1285 fs_manager.cc:359] Metadata directory not provided
I20260925 15:18:15.835345 1285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:18:15.835580 1285 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:18:15.838383 1285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349162370577-12320-0/minicluster-data/master-0/data/instance:
uuid: "139cce44f6af4bbeb02902dfc4cb15e1"
format_stamp: "Formatted at 2026-09-25 15:18:15 on dist-test-slave-p468"
server_key: "4f98b31f59bffbf510cabb32b10470a2"
server_key_iv: "dd53083c7d3e48c28a699130772b4db5"
server_key_version: "encryptionkey@0"
I20260925 15:18:15.839102 1285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349162370577-12320-0/minicluster-data/master-0/wal/instance:
uuid: "139cce44f6af4bbeb02902dfc4cb15e1"
format_stamp: "Formatted at 2026-09-25 15:18:15 on dist-test-slave-p468"
server_key: "4f98b31f59bffbf510cabb32b10470a2"
server_key_iv: "dd53083c7d3e48c28a699130772b4db5"
server_key_version: "encryptionkey@0"
I20260925 15:18:15.843796 1285 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 15:18:15.846997 1300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:15.848284 1285 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:18:15.848455 1285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349162370577-12320-0/minicluster-data/master-0/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349162370577-12320-0/minicluster-data/master-0/wal
uuid: "139cce44f6af4bbeb02902dfc4cb15e1"
format_stamp: "Formatted at 2026-09-25 15:18:15 on dist-test-slave-p468"
server_key: "4f98b31f59bffbf510cabb32b10470a2"
server_key_iv: "dd53083c7d3e48c28a699130772b4db5"
server_key_version: "encryptionkey@0"
I20260925 15:18:15.848641 1285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349162370577-12320-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349162370577-12320-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349162370577-12320-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:18:15.889604 1285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:18:15.893950 1285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:18:15.894333 1285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:18:15.914361 1285 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38813
I20260925 15:18:15.914407 1351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38813 every 8 connection(s)
I20260925 15:18:15.915794 1285 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
I20260925 15:18:15.919972 1352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:18:15.920573 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 1285
I20260925 15:18:15.920768 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349162370577-12320-0/minicluster-data/master-0/wal/instance
I20260925 15:18:15.921342 12320 external_mini_cluster.cc:1428] Setting key 65b299357395d1df3ae091189b2e5a88
I20260925 15:18:15.932739 1352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 139cce44f6af4bbeb02902dfc4cb15e1: Bootstrap starting.
I20260925 15:18:15.935724 1355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:15.923174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.242:46303 (local address 127.0.0.1:38813)
0925 15:18:15.923891 (+ 717us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:15.923901 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:15.923939 (+ 38us) server_negotiation.cc:407] Connection header received
0925 15:18:15.926563 (+ 2624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:15.926587 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:15.927118 (+ 531us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:15.927548 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:15.928124 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:15.929490 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:15.931059 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:15.931775 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:15.931979 (+ 204us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:15.933396 (+ 1417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:15.933404 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:15.933418 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:18:15.933961 (+ 543us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:15.934057 (+ 96us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:18:15.934090 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:15.934312 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:15.934598 (+ 286us) server_negotiation.cc:299] Negotiation successful
0925 15:18:15.934769 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":306,"threads_started":1}
I20260925 15:18:15.935940 1352 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 139cce44f6af4bbeb02902dfc4cb15e1: Neither blocks nor log segments found. Creating new log.
I20260925 15:18:15.937032 1352 log.cc:824] T 00000000000000000000000000000000 P 139cce44f6af4bbeb02902dfc4cb15e1: Log is configured to *not* fsync() on all Append() calls
I20260925 15:18:15.940058 12320 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:18:15.940500 1352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 139cce44f6af4bbeb02902dfc4cb15e1: No bootstrap required, opened a new log
I20260925 15:18:15.950495 1352 raft_consensus.cc:348] T 00000000000000000000000000000000 P 139cce44f6af4bbeb02902dfc4cb15e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "139cce44f6af4bbeb02902dfc4cb15e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38813 } }
I20260925 15:18:15.950906 1352 raft_consensus.cc:374] T 00000000000000000000000000000000 P 139cce44f6af4bbeb02902dfc4cb15e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:18:15.951038 1352 raft_consensus.cc:729] T 00000000000000000000000000000000 P 139cce44f6af4bbeb02902dfc4cb15e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 139cce44f6af4bbeb02902dfc4cb15e1, State: Initialized, Role: FOLLOWER
I20260925 15:18:15.951758 1352 consensus_queue.cc:260] T 00000000000000000000000000000000 P 139cce44f6af4bbeb02902dfc4cb15e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "139cce44f6af4bbeb02902dfc4cb15e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38813 } }
I20260925 15:18:15.952106 1352 raft_consensus.cc:388] T 00000000000000000000000000000000 P 139cce44f6af4bbeb02902dfc4cb15e1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:18:15.952154 1355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:15.945470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.242:43627 (local address 127.0.0.1:38813)
0925 15:18:15.945701 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:15.945712 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:15.945983 (+ 271us) server_negotiation.cc:407] Connection header received
0925 15:18:15.946152 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:15.946160 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:15.946267 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:15.946382 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:15.946820 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:15.948006 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:15.949364 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:15.949995 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:15.950137 (+ 142us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:15.951328 (+ 1191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:15.951333 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:15.951336 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:18:15.951354 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:15.951391 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:18:15.951410 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:15.951495 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:15.951713 (+ 218us) server_negotiation.cc:299] Negotiation successful
0925 15:18:15.951788 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 15:18:15.952307 1352 raft_consensus.cc:482] T 00000000000000000000000000000000 P 139cce44f6af4bbeb02902dfc4cb15e1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:18:15.952504 1352 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 139cce44f6af4bbeb02902dfc4cb15e1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:18:15.955037 1352 raft_consensus.cc:504] T 00000000000000000000000000000000 P 139cce44f6af4bbeb02902dfc4cb15e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "139cce44f6af4bbeb02902dfc4cb15e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38813 } }
I20260925 15:18:15.955476 1352 leader_election.cc:302] T 00000000000000000000000000000000 P 139cce44f6af4bbeb02902dfc4cb15e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 139cce44f6af4bbeb02902dfc4cb15e1; no voters:
I20260925 15:18:15.956103 1352 leader_election.cc:288] T 00000000000000000000000000000000 P 139cce44f6af4bbeb02902dfc4cb15e1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:18:15.956202 1362 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 139cce44f6af4bbeb02902dfc4cb15e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:18:15.956835 1362 raft_consensus.cc:686] T 00000000000000000000000000000000 P 139cce44f6af4bbeb02902dfc4cb15e1 [term 1 LEADER]: Becoming Leader. State: Replica: 139cce44f6af4bbeb02902dfc4cb15e1, State: Running, Role: LEADER
I20260925 15:18:15.957438 1362 consensus_queue.cc:237] T 00000000000000000000000000000000 P 139cce44f6af4bbeb02902dfc4cb15e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "139cce44f6af4bbeb02902dfc4cb15e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38813 } }
I20260925 15:18:15.958338 1352 sys_catalog.cc:564] T 00000000000000000000000000000000 P 139cce44f6af4bbeb02902dfc4cb15e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:18:15.960328 1363 sys_catalog.cc:455] T 00000000000000000000000000000000 P 139cce44f6af4bbeb02902dfc4cb15e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "139cce44f6af4bbeb02902dfc4cb15e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "139cce44f6af4bbeb02902dfc4cb15e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38813 } } }
I20260925 15:18:15.960700 1363 sys_catalog.cc:458] T 00000000000000000000000000000000 P 139cce44f6af4bbeb02902dfc4cb15e1 [sys.catalog]: This master's current role is: LEADER
I20260925 15:18:15.960530 1364 sys_catalog.cc:455] T 00000000000000000000000000000000 P 139cce44f6af4bbeb02902dfc4cb15e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 139cce44f6af4bbeb02902dfc4cb15e1. Latest consensus state: current_term: 1 leader_uuid: "139cce44f6af4bbeb02902dfc4cb15e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "139cce44f6af4bbeb02902dfc4cb15e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38813 } } }
I20260925 15:18:15.960901 1364 sys_catalog.cc:458] T 00000000000000000000000000000000 P 139cce44f6af4bbeb02902dfc4cb15e1 [sys.catalog]: This master's current role is: LEADER
I20260925 15:18:15.962419 1369 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:18:15.968082 1369 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:18:15.976655 1369 catalog_manager.cc:1347] Generated new cluster ID: 1f308efb75fb47dba744118bbe5ecd30
I20260925 15:18:15.976764 1369 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:18:15.999647 1369 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:18:16.001535 1369 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:18:16.012048 1369 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 139cce44f6af4bbeb02902dfc4cb15e1: Generated new TSK 0
I20260925 15:18:16.013082 1369 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:18:16.029588 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 1285
W20260925 15:18:16.030766 1347 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.242:46303
W20260925 15:18:16.031695 12320 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 1285: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 15:18:16.032006 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 1285
W20260925 15:18:16.049253 1270 connection.cc:535] client connection to 127.0.0.1:38813 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38813): Connection reset by peer (error 104)
2026-09-25T15:18:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1939 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349162370577-12320-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:18:16 dist-test-slave-p468 krb5kdc[1383](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:18:16 dist-test-slave-p468 krb5kdc[1383](info): set up 2 sockets
Sep 25 15:18:16 dist-test-slave-p468 krb5kdc[1383](info): commencing operation
krb5kdc: starting...
W20260925 15:18:17.170584 12320 mini_kdc.cc:121] Time spent starting KDC: real 1.102s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349162370577-12320-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349162370577-12320-0/krb5kdc/test-user.keytab.
Sep 25 15:18:17 dist-test-slave-p468 krb5kdc[1383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349497, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:18:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:18:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349162370577-12320-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349162370577-12320-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:18:17.422789 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349162370577-12320-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349162370577-12320-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349162370577-12320-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45237
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349162370577-12320-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45237
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:18:17.747562 1396 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:18:17.747844 1396 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:18:17.748010 1396 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:18:17.755300 1396 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:18:17.755410 1396 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:18:17.755519 1396 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:18:17.755625 1396 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:18:17.755702 1396 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:18:17.765555 1396 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349162370577-12320-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349162370577-12320-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45237
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349162370577-12320-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45237
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349162370577-12320-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:18:17.766981 1396 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:18:17.768307 1396 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:18:17.780325 1404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:17.780397 1401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:17.780272 1402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:18:17.780772 1396 server_base.cc:1034] running on GCE node
I20260925 15:18:17.781687 1396 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:18:17.783207 1396 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:18:17.784409 1396 hybrid_clock.cc:648] HybridClock initialized: now 1790349497784368 us; error 34 us; skew 500 ppm
I20260925 15:18:17.784782 1396 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:18:17 dist-test-slave-p468 krb5kdc[1383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349497, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:18:17.790992 1396 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:18:17.792418 1396 webserver.cc:460] Webserver started at http://127.0.0.1:42213/ using document root <none> and password file <none>
I20260925 15:18:17.793081 1396 fs_manager.cc:359] Metadata directory not provided
I20260925 15:18:17.793228 1396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:18:17.793526 1396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:18:17.796460 1396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349162370577-12320-0/minicluster-data/master-0/data/instance:
uuid: "08fa31fb1a534e21a509fed8ab33cd83"
format_stamp: "Formatted at 2026-09-25 15:18:17 on dist-test-slave-p468"
server_key: "9c0a00000a6e14a6b2cf1e579415dc1c"
server_key_iv: "6c601cbf85a0f047322871ee00d65280"
server_key_version: "encryptionkey@0"
I20260925 15:18:17.797214 1396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349162370577-12320-0/minicluster-data/master-0/wal/instance:
uuid: "08fa31fb1a534e21a509fed8ab33cd83"
format_stamp: "Formatted at 2026-09-25 15:18:17 on dist-test-slave-p468"
server_key: "9c0a00000a6e14a6b2cf1e579415dc1c"
server_key_iv: "6c601cbf85a0f047322871ee00d65280"
server_key_version: "encryptionkey@0"
I20260925 15:18:17.802424 1396 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 15:18:17.805733 1411 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:17.807071 1396 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:18:17.807253 1396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349162370577-12320-0/minicluster-data/master-0/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349162370577-12320-0/minicluster-data/master-0/wal
uuid: "08fa31fb1a534e21a509fed8ab33cd83"
format_stamp: "Formatted at 2026-09-25 15:18:17 on dist-test-slave-p468"
server_key: "9c0a00000a6e14a6b2cf1e579415dc1c"
server_key_iv: "6c601cbf85a0f047322871ee00d65280"
server_key_version: "encryptionkey@0"
I20260925 15:18:17.807451 1396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349162370577-12320-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349162370577-12320-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349162370577-12320-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:18:17.813004 1396 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:18:17.817454 1396 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:18:17.817770 1396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:18:17.836199 1396 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45237
I20260925 15:18:17.836339 1462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45237 every 8 connection(s)
I20260925 15:18:17.837594 1396 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
I20260925 15:18:17.841413 1463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:18:17.846717 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 1396
I20260925 15:18:17.846882 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349162370577-12320-0/minicluster-data/master-0/wal/instance
I20260925 15:18:17.847419 12320 external_mini_cluster.cc:1428] Setting key b6202a2a20443e8c98e5347dbe3ff636
I20260925 15:18:17.852581 1463 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08fa31fb1a534e21a509fed8ab33cd83: Bootstrap starting.
I20260925 15:18:17.855268 1466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:17.849307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.242:51803 (local address 127.0.0.1:45237)
0925 15:18:17.850117 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:17.850128 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:17.850172 (+ 44us) server_negotiation.cc:407] Connection header received
0925 15:18:17.851900 (+ 1728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:17.851921 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:17.852255 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:17.852631 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:17.853607 (+ 976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:17.853613 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:17.853630 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:18:17.853648 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:17.853698 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:18:17.853720 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:17.853967 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:17.854122 (+ 155us) server_negotiation.cc:299] Negotiation successful
0925 15:18:17.854347 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":310,"threads_started":1}
I20260925 15:18:17.856149 1463 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 08fa31fb1a534e21a509fed8ab33cd83: Neither blocks nor log segments found. Creating new log.
I20260925 15:18:17.857282 1463 log.cc:824] T 00000000000000000000000000000000 P 08fa31fb1a534e21a509fed8ab33cd83: Log is configured to *not* fsync() on all Append() calls
I20260925 15:18:17.859505 12320 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:18:17.860930 1463 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08fa31fb1a534e21a509fed8ab33cd83: No bootstrap required, opened a new log
I20260925 15:18:17.868878 1466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:17.865314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.242:37417 (local address 127.0.0.1:45237)
0925 15:18:17.865505 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:17.865511 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:17.866665 (+ 1154us) server_negotiation.cc:407] Connection header received
0925 15:18:17.866839 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:17.866844 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:17.866908 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:17.867020 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:17.868043 (+ 1023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:17.868049 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:17.868054 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:18:17.868072 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:17.868123 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:18:17.868143 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:17.868273 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:17.868485 (+ 212us) server_negotiation.cc:299] Negotiation successful
0925 15:18:17.868579 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 15:18:17.871924 1463 raft_consensus.cc:348] T 00000000000000000000000000000000 P 08fa31fb1a534e21a509fed8ab33cd83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08fa31fb1a534e21a509fed8ab33cd83" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45237 } }
I20260925 15:18:17.872349 1463 raft_consensus.cc:374] T 00000000000000000000000000000000 P 08fa31fb1a534e21a509fed8ab33cd83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:18:17.872453 1463 raft_consensus.cc:729] T 00000000000000000000000000000000 P 08fa31fb1a534e21a509fed8ab33cd83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08fa31fb1a534e21a509fed8ab33cd83, State: Initialized, Role: FOLLOWER
I20260925 15:18:17.872949 1463 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08fa31fb1a534e21a509fed8ab33cd83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08fa31fb1a534e21a509fed8ab33cd83" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45237 } }
I20260925 15:18:17.873150 1463 raft_consensus.cc:388] T 00000000000000000000000000000000 P 08fa31fb1a534e21a509fed8ab33cd83 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:18:17.873245 1463 raft_consensus.cc:482] T 00000000000000000000000000000000 P 08fa31fb1a534e21a509fed8ab33cd83 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:18:17.873360 1463 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08fa31fb1a534e21a509fed8ab33cd83 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:18:17.875543 1463 raft_consensus.cc:504] T 00000000000000000000000000000000 P 08fa31fb1a534e21a509fed8ab33cd83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08fa31fb1a534e21a509fed8ab33cd83" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45237 } }
I20260925 15:18:17.875972 1463 leader_election.cc:302] T 00000000000000000000000000000000 P 08fa31fb1a534e21a509fed8ab33cd83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 08fa31fb1a534e21a509fed8ab33cd83; no voters:
I20260925 15:18:17.876499 1463 leader_election.cc:288] T 00000000000000000000000000000000 P 08fa31fb1a534e21a509fed8ab33cd83 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:18:17.876663 1473 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 08fa31fb1a534e21a509fed8ab33cd83 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:18:17.878044 1473 raft_consensus.cc:686] T 00000000000000000000000000000000 P 08fa31fb1a534e21a509fed8ab33cd83 [term 1 LEADER]: Becoming Leader. State: Replica: 08fa31fb1a534e21a509fed8ab33cd83, State: Running, Role: LEADER
I20260925 15:18:17.878545 1473 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08fa31fb1a534e21a509fed8ab33cd83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08fa31fb1a534e21a509fed8ab33cd83" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45237 } }
I20260925 15:18:17.878760 1463 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08fa31fb1a534e21a509fed8ab33cd83 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:18:17.881424 1475 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08fa31fb1a534e21a509fed8ab33cd83 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08fa31fb1a534e21a509fed8ab33cd83. Latest consensus state: current_term: 1 leader_uuid: "08fa31fb1a534e21a509fed8ab33cd83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08fa31fb1a534e21a509fed8ab33cd83" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45237 } } }
I20260925 15:18:17.882102 1475 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08fa31fb1a534e21a509fed8ab33cd83 [sys.catalog]: This master's current role is: LEADER
I20260925 15:18:17.881526 1474 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08fa31fb1a534e21a509fed8ab33cd83 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "08fa31fb1a534e21a509fed8ab33cd83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08fa31fb1a534e21a509fed8ab33cd83" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45237 } } }
I20260925 15:18:17.882704 1474 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08fa31fb1a534e21a509fed8ab33cd83 [sys.catalog]: This master's current role is: LEADER
I20260925 15:18:17.883296 1482 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:18:17.888685 1482 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:18:17.897253 1482 catalog_manager.cc:1347] Generated new cluster ID: 1ca9659d2d38492384f10b65e41d30ab
I20260925 15:18:17.897361 1482 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:18:17.915060 1482 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:18:17.915493 1482 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:18:17.924281 1482 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 08fa31fb1a534e21a509fed8ab33cd83: Generated new TSK 0
I20260925 15:18:17.925042 1482 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:18:17.935774 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 1396
W20260925 15:18:17.936940 1458 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.242:51803
W20260925 15:18:17.937636 12320 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 1396: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 15:18:17.937793 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 1396
2026-09-25T15:18:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1904 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349162370577-12320-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:18:18 dist-test-slave-p468 krb5kdc[1494](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:18:18 dist-test-slave-p468 krb5kdc[1494](info): set up 2 sockets
Sep 25 15:18:18 dist-test-slave-p468 krb5kdc[1494](info): commencing operation
krb5kdc: starting...
W20260925 15:18:19.047108 12320 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.002s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349162370577-12320-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349162370577-12320-0/krb5kdc/test-user.keytab.
Sep 25 15:18:19 dist-test-slave-p468 krb5kdc[1494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349499, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:18:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:18:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.8.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab.
Entry for principal kudu/127.12.8.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab.
I20260925 15:18:19.293386 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349162370577-12320-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349162370577-12320-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349162370577-12320-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.62:39921
--webserver_interface=127.12.8.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab
--principal=kudu/127.12.8.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:46613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.62:39921
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:18:19.615386 1507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:18:19.615716 1507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:18:19.615864 1507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:18:19.623059 1507 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:18:19.623202 1507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:18:19.623301 1507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:18:19.623404 1507 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:18:19.623474 1507 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:18:19.633096 1507 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:46613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349162370577-12320-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349162370577-12320-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.8.62:39921
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab
--principal=kudu/127.12.8.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.62:39921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.8.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349162370577-12320-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:18:19.634449 1507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:18:19.635763 1507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:18:19.647301 1512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:19.647248 1513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:19.647326 1515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:18:19.648101 1507 server_base.cc:1034] running on GCE node
I20260925 15:18:19.648833 1507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:18:19.650161 1507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:18:19.651422 1507 hybrid_clock.cc:648] HybridClock initialized: now 1790349499651373 us; error 46 us; skew 500 ppm
I20260925 15:18:19.651762 1507 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:18:19 dist-test-slave-p468 krb5kdc[1494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349499, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:18:19.654873 1507 init.cc:377] Logged in from keytab as kudu/127.12.8.62@KRBTEST.COM (short username kudu)
I20260925 15:18:19.656397 1507 webserver.cc:460] Webserver started at http://127.12.8.62:35525/ using document root <none> and password file <none>
I20260925 15:18:19.657068 1507 fs_manager.cc:359] Metadata directory not provided
I20260925 15:18:19.657204 1507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:18:19.657507 1507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:18:19.660566 1507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349162370577-12320-0/minicluster-data/master-0/data/instance:
uuid: "a0005b7e6b86420ca7780ee8300de43d"
format_stamp: "Formatted at 2026-09-25 15:18:19 on dist-test-slave-p468"
server_key: "eff71b64ef69b6d7368320d7e04eccb1"
server_key_iv: "5a2f7511b397fef8597792a203f0c6b4"
server_key_version: "encryptionkey@0"
I20260925 15:18:19.661367 1507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349162370577-12320-0/minicluster-data/master-0/wal/instance:
uuid: "a0005b7e6b86420ca7780ee8300de43d"
format_stamp: "Formatted at 2026-09-25 15:18:19 on dist-test-slave-p468"
server_key: "eff71b64ef69b6d7368320d7e04eccb1"
server_key_iv: "5a2f7511b397fef8597792a203f0c6b4"
server_key_version: "encryptionkey@0"
I20260925 15:18:19.666595 1507 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 15:18:19.670022 1522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:19.671435 1507 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:18:19.671614 1507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349162370577-12320-0/minicluster-data/master-0/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349162370577-12320-0/minicluster-data/master-0/wal
uuid: "a0005b7e6b86420ca7780ee8300de43d"
format_stamp: "Formatted at 2026-09-25 15:18:19 on dist-test-slave-p468"
server_key: "eff71b64ef69b6d7368320d7e04eccb1"
server_key_iv: "5a2f7511b397fef8597792a203f0c6b4"
server_key_version: "encryptionkey@0"
I20260925 15:18:19.671840 1507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349162370577-12320-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349162370577-12320-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349162370577-12320-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:18:19.685999 1507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:18:19.689775 1507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:18:19.690043 1507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:18:19.708356 1507 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.62:39921
I20260925 15:18:19.708410 1573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.62:39921 every 8 connection(s)
I20260925 15:18:19.709619 1507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
I20260925 15:18:19.713399 1574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:18:19.716151 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 1507
I20260925 15:18:19.716354 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349162370577-12320-0/minicluster-data/master-0/wal/instance
I20260925 15:18:19.716933 12320 external_mini_cluster.cc:1428] Setting key c5dd314ec5439cfd1ca90afdca64e69b
I20260925 15:18:19.726735 1574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0005b7e6b86420ca7780ee8300de43d: Bootstrap starting.
I20260925 15:18:19.730967 1574 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a0005b7e6b86420ca7780ee8300de43d: Neither blocks nor log segments found. Creating new log.
Sep 25 15:18:19 dist-test-slave-p468 krb5kdc[1494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349499, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
I20260925 15:18:19.732214 1574 log.cc:824] T 00000000000000000000000000000000 P a0005b7e6b86420ca7780ee8300de43d: Log is configured to *not* fsync() on all Append() calls
I20260925 15:18:19.735773 1574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0005b7e6b86420ca7780ee8300de43d: No bootstrap required, opened a new log
I20260925 15:18:19.742168 1577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:19.718722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35695 (local address 127.12.8.62:39921)
0925 15:18:19.719526 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:19.719539 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:19.719581 (+ 42us) server_negotiation.cc:407] Connection header received
0925 15:18:19.721198 (+ 1617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:19.721218 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:19.721589 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:19.721887 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:19.723808 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:19.725126 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:19.727112 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:19.727788 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:19.727913 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:19.733081 (+ 5168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:19.733089 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:19.733102 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:19.733150 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:19.737302 (+ 4152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:19.738036 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:19.738045 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:19.738054 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:19.738317 (+ 263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:19.738767 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:19.738773 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:19.738777 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:19.739559 (+ 782us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:19.739949 (+ 390us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:19.740319 (+ 370us) server_negotiation.cc:299] Negotiation successful
0925 15:18:19.740698 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":625,"thread_start_us":350,"threads_started":1}
I20260925 15:18:19.746106 12320 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:18:19.747087 1574 raft_consensus.cc:348] T 00000000000000000000000000000000 P a0005b7e6b86420ca7780ee8300de43d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0005b7e6b86420ca7780ee8300de43d" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 39921 } }
I20260925 15:18:19.747453 1574 raft_consensus.cc:374] T 00000000000000000000000000000000 P a0005b7e6b86420ca7780ee8300de43d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:18:19.747574 1574 raft_consensus.cc:729] T 00000000000000000000000000000000 P a0005b7e6b86420ca7780ee8300de43d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0005b7e6b86420ca7780ee8300de43d, State: Initialized, Role: FOLLOWER
I20260925 15:18:19.748129 1574 consensus_queue.cc:260] T 00000000000000000000000000000000 P a0005b7e6b86420ca7780ee8300de43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0005b7e6b86420ca7780ee8300de43d" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 39921 } }
I20260925 15:18:19.748378 1574 raft_consensus.cc:388] T 00000000000000000000000000000000 P a0005b7e6b86420ca7780ee8300de43d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:18:19.748555 1574 raft_consensus.cc:482] T 00000000000000000000000000000000 P a0005b7e6b86420ca7780ee8300de43d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:18:19.748729 1574 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a0005b7e6b86420ca7780ee8300de43d [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:18:19.751335 1574 raft_consensus.cc:504] T 00000000000000000000000000000000 P a0005b7e6b86420ca7780ee8300de43d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0005b7e6b86420ca7780ee8300de43d" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 39921 } }
I20260925 15:18:19.751928 1574 leader_election.cc:302] T 00000000000000000000000000000000 P a0005b7e6b86420ca7780ee8300de43d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a0005b7e6b86420ca7780ee8300de43d; no voters:
I20260925 15:18:19.752503 1574 leader_election.cc:288] T 00000000000000000000000000000000 P a0005b7e6b86420ca7780ee8300de43d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:18:19.752664 1584 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a0005b7e6b86420ca7780ee8300de43d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:18:19.754706 1574 sys_catalog.cc:564] T 00000000000000000000000000000000 P a0005b7e6b86420ca7780ee8300de43d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:18:19.755411 1584 raft_consensus.cc:686] T 00000000000000000000000000000000 P a0005b7e6b86420ca7780ee8300de43d [term 1 LEADER]: Becoming Leader. State: Replica: a0005b7e6b86420ca7780ee8300de43d, State: Running, Role: LEADER
I20260925 15:18:19.755929 1584 consensus_queue.cc:237] T 00000000000000000000000000000000 P a0005b7e6b86420ca7780ee8300de43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0005b7e6b86420ca7780ee8300de43d" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 39921 } }
I20260925 15:18:19.758908 1586 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0005b7e6b86420ca7780ee8300de43d [sys.catalog]: SysCatalogTable state changed. Reason: New leader a0005b7e6b86420ca7780ee8300de43d. Latest consensus state: current_term: 1 leader_uuid: "a0005b7e6b86420ca7780ee8300de43d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0005b7e6b86420ca7780ee8300de43d" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 39921 } } }
I20260925 15:18:19.759311 1586 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0005b7e6b86420ca7780ee8300de43d [sys.catalog]: This master's current role is: LEADER
I20260925 15:18:19.759322 1585 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0005b7e6b86420ca7780ee8300de43d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a0005b7e6b86420ca7780ee8300de43d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0005b7e6b86420ca7780ee8300de43d" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 39921 } } }
I20260925 15:18:19.760349 1585 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0005b7e6b86420ca7780ee8300de43d [sys.catalog]: This master's current role is: LEADER
I20260925 15:18:19.760416 1593 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:18:19.769001 1593 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:18:19.772266 1577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:19.750510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55563 (local address 127.12.8.62:39921)
0925 15:18:19.750731 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:19.750740 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:19.751278 (+ 538us) server_negotiation.cc:407] Connection header received
0925 15:18:19.751412 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:19.751418 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:19.751500 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:19.751607 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:19.753457 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:19.754979 (+ 1522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:19.757282 (+ 2303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:19.758372 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:19.758550 (+ 178us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:19.763022 (+ 4472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:19.763032 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:19.763040 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:19.763091 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:19.767858 (+ 4767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:19.768866 (+ 1008us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:19.768873 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:19.768876 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:19.769039 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:19.769592 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:19.769597 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:19.769602 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:19.770075 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:19.770384 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:19.771653 (+ 1269us) server_negotiation.cc:299] Negotiation successful
0925 15:18:19.771849 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 15:18:19.781422 1593 catalog_manager.cc:1347] Generated new cluster ID: 668459cc3bf4462a83791ee7e56b4f72
I20260925 15:18:19.781545 1593 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:18:19.796082 1593 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:18:19.797510 1593 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:18:19.811252 1593 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a0005b7e6b86420ca7780ee8300de43d: Generated new TSK 0
I20260925 15:18:19.812098 1593 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:18:19.824414 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 1507
I20260925 15:18:19.825634 1569 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:18:19.922533 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 1507
W20260925 15:18:19.936203 1492 connection.cc:535] client connection to 127.12.8.62:39921 recv error: Network error: failed to read from TLS socket (remote: 127.12.8.62:39921): Connection reset by peer (error 104)
2026-09-25T15:18:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1982 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349162370577-12320-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:18:20 dist-test-slave-p468 krb5kdc[1606](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:18:20 dist-test-slave-p468 krb5kdc[1606](info): set up 2 sockets
Sep 25 15:18:20 dist-test-slave-p468 krb5kdc[1606](info): commencing operation
krb5kdc: starting...
W20260925 15:18:21.106827 12320 mini_kdc.cc:121] Time spent starting KDC: real 1.151s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349162370577-12320-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349162370577-12320-0/krb5kdc/test-user.keytab.
Sep 25 15:18:21 dist-test-slave-p468 krb5kdc[1606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349501, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:18:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:18:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.8.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab.
Entry for principal kudu/127.12.8.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab.
I20260925 15:18:21.357261 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349162370577-12320-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349162370577-12320-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349162370577-12320-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.62:40255
--webserver_interface=127.12.8.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab
--principal=kudu/127.12.8.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:32963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.62:40255
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:18:21.676793 1619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:18:21.677054 1619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:18:21.677217 1619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:18:21.684617 1619 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:18:21.684728 1619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:18:21.684856 1619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:18:21.684966 1619 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:18:21.685040 1619 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:18:21.695171 1619 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:32963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349162370577-12320-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349162370577-12320-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.8.62:40255
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab
--principal=kudu/127.12.8.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.62:40255
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.8.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349162370577-12320-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:18:21.696496 1619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:18:21.697804 1619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:18:21.709564 1624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:21.709626 1627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:21.709589 1625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:18:21.710844 1619 server_base.cc:1034] running on GCE node
I20260925 15:18:21.711671 1619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:18:21.713085 1619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:18:21.714290 1619 hybrid_clock.cc:648] HybridClock initialized: now 1790349501714270 us; error 87 us; skew 500 ppm
I20260925 15:18:21.714633 1619 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:18:21 dist-test-slave-p468 krb5kdc[1606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349501, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:18:21.717770 1619 init.cc:377] Logged in from keytab as kudu/127.12.8.62@KRBTEST.COM (short username kudu)
I20260925 15:18:21.719302 1619 webserver.cc:460] Webserver started at http://127.12.8.62:33601/ using document root <none> and password file <none>
I20260925 15:18:21.719955 1619 fs_manager.cc:359] Metadata directory not provided
I20260925 15:18:21.720093 1619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:18:21.720386 1619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:18:21.723366 1619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349162370577-12320-0/minicluster-data/master-0/data/instance:
uuid: "8993e377426a46ccae4aaf9ab527298e"
format_stamp: "Formatted at 2026-09-25 15:18:21 on dist-test-slave-p468"
server_key: "4dd39d972b7681e5f11ee18441174f8b"
server_key_iv: "9d76b2ef28dc95a6a50eb3ec28218307"
server_key_version: "encryptionkey@0"
I20260925 15:18:21.724107 1619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349162370577-12320-0/minicluster-data/master-0/wal/instance:
uuid: "8993e377426a46ccae4aaf9ab527298e"
format_stamp: "Formatted at 2026-09-25 15:18:21 on dist-test-slave-p468"
server_key: "4dd39d972b7681e5f11ee18441174f8b"
server_key_iv: "9d76b2ef28dc95a6a50eb3ec28218307"
server_key_version: "encryptionkey@0"
I20260925 15:18:21.729110 1619 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260925 15:18:21.732534 1634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:21.733862 1619 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 15:18:21.734031 1619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349162370577-12320-0/minicluster-data/master-0/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349162370577-12320-0/minicluster-data/master-0/wal
uuid: "8993e377426a46ccae4aaf9ab527298e"
format_stamp: "Formatted at 2026-09-25 15:18:21 on dist-test-slave-p468"
server_key: "4dd39d972b7681e5f11ee18441174f8b"
server_key_iv: "9d76b2ef28dc95a6a50eb3ec28218307"
server_key_version: "encryptionkey@0"
I20260925 15:18:21.734246 1619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349162370577-12320-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349162370577-12320-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349162370577-12320-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:18:21.746951 1619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:18:21.751230 1619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:18:21.751502 1619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:18:21.769754 1619 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.62:40255
I20260925 15:18:21.769802 1685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.62:40255 every 8 connection(s)
I20260925 15:18:21.771185 1619 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
I20260925 15:18:21.774803 1686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:18:21.781038 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 1619
I20260925 15:18:21.781304 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349162370577-12320-0/minicluster-data/master-0/wal/instance
I20260925 15:18:21.781778 12320 external_mini_cluster.cc:1428] Setting key 67f9b7bd015cabcfdb34cbae6b3d65a1
I20260925 15:18:21.786518 1686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8993e377426a46ccae4aaf9ab527298e: Bootstrap starting.
I20260925 15:18:21.790391 1686 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8993e377426a46ccae4aaf9ab527298e: Neither blocks nor log segments found. Creating new log.
I20260925 15:18:21.791383 1686 log.cc:824] T 00000000000000000000000000000000 P 8993e377426a46ccae4aaf9ab527298e: Log is configured to *not* fsync() on all Append() calls
I20260925 15:18:21.794376 1686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8993e377426a46ccae4aaf9ab527298e: No bootstrap required, opened a new log
I20260925 15:18:21.796710 1689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:21.783759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49669 (local address 127.12.8.62:40255)
0925 15:18:21.784555 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:21.784569 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:21.784615 (+ 46us) server_negotiation.cc:407] Connection header received
0925 15:18:21.786944 (+ 2329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:21.786969 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:21.787465 (+ 496us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:21.787871 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:21.788439 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:21.790402 (+ 1963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:21.791872 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:21.792679 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:21.792869 (+ 190us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:21.793971 (+ 1102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:21.793981 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:21.793998 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:18:21.794706 (+ 708us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:21.794802 (+ 96us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:18:21.794838 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:21.795012 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:21.795312 (+ 300us) server_negotiation.cc:299] Negotiation successful
0925 15:18:21.795467 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":352,"threads_started":1}
I20260925 15:18:21.800915 12320 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:18:21.803001 1686 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8993e377426a46ccae4aaf9ab527298e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8993e377426a46ccae4aaf9ab527298e" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 40255 } }
I20260925 15:18:21.803360 1686 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8993e377426a46ccae4aaf9ab527298e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:18:21.803471 1686 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8993e377426a46ccae4aaf9ab527298e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8993e377426a46ccae4aaf9ab527298e, State: Initialized, Role: FOLLOWER
I20260925 15:18:21.804082 1686 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8993e377426a46ccae4aaf9ab527298e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8993e377426a46ccae4aaf9ab527298e" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 40255 } }
I20260925 15:18:21.804346 1686 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8993e377426a46ccae4aaf9ab527298e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:18:21.804498 1686 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8993e377426a46ccae4aaf9ab527298e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:18:21.804689 1686 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8993e377426a46ccae4aaf9ab527298e [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:18:21.807758 1686 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8993e377426a46ccae4aaf9ab527298e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8993e377426a46ccae4aaf9ab527298e" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 40255 } }
I20260925 15:18:21.808246 1686 leader_election.cc:302] T 00000000000000000000000000000000 P 8993e377426a46ccae4aaf9ab527298e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8993e377426a46ccae4aaf9ab527298e; no voters:
I20260925 15:18:21.808913 1686 leader_election.cc:288] T 00000000000000000000000000000000 P 8993e377426a46ccae4aaf9ab527298e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:18:21.809007 1696 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8993e377426a46ccae4aaf9ab527298e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:18:21.811508 1686 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8993e377426a46ccae4aaf9ab527298e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:18:21.812354 1689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:21.805458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37753 (local address 127.12.8.62:40255)
0925 15:18:21.805671 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:21.805680 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:21.806126 (+ 446us) server_negotiation.cc:407] Connection header received
0925 15:18:21.806336 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:21.806340 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:21.806405 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:21.806489 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:21.807108 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:21.807871 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:21.809463 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:21.810116 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:21.810236 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:21.811524 (+ 1288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:21.811530 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:21.811534 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:18:21.811559 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:21.811632 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:18:21.811656 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:21.811774 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:21.811962 (+ 188us) server_negotiation.cc:299] Negotiation successful
0925 15:18:21.812015 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 15:18:21.814476 1696 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8993e377426a46ccae4aaf9ab527298e [term 1 LEADER]: Becoming Leader. State: Replica: 8993e377426a46ccae4aaf9ab527298e, State: Running, Role: LEADER
I20260925 15:18:21.815204 1696 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8993e377426a46ccae4aaf9ab527298e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8993e377426a46ccae4aaf9ab527298e" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 40255 } }
I20260925 15:18:21.818475 1697 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8993e377426a46ccae4aaf9ab527298e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8993e377426a46ccae4aaf9ab527298e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8993e377426a46ccae4aaf9ab527298e" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 40255 } } }
I20260925 15:18:21.818578 1698 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8993e377426a46ccae4aaf9ab527298e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8993e377426a46ccae4aaf9ab527298e. Latest consensus state: current_term: 1 leader_uuid: "8993e377426a46ccae4aaf9ab527298e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8993e377426a46ccae4aaf9ab527298e" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 40255 } } }
I20260925 15:18:21.818991 1698 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8993e377426a46ccae4aaf9ab527298e [sys.catalog]: This master's current role is: LEADER
I20260925 15:18:21.818876 1697 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8993e377426a46ccae4aaf9ab527298e [sys.catalog]: This master's current role is: LEADER
I20260925 15:18:21.822985 1705 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:18:21.828200 1705 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:18:21.836640 1705 catalog_manager.cc:1347] Generated new cluster ID: 565acff680a74bd9a75a4a55258fb942
I20260925 15:18:21.836745 1705 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:18:21.850651 1705 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:18:21.852421 1705 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:18:21.885247 1705 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8993e377426a46ccae4aaf9ab527298e: Generated new TSK 0
I20260925 15:18:21.886523 1705 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:18:21.901400 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 1619
W20260925 15:18:21.903164 1681 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:49669
W20260925 15:18:21.904062 12320 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 1619: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 15:18:21.904217 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 1619
W20260925 15:18:21.917634 1604 connection.cc:535] client connection to 127.12.8.62:40255 recv error: Network error: failed to read from TLS socket (remote: 127.12.8.62:40255): Connection reset by peer (error 104)
2026-09-25T15:18:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1980 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349162370577-12320-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:18:21 dist-test-slave-p468 krb5kdc[1717](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:18:21 dist-test-slave-p468 krb5kdc[1717](info): set up 2 sockets
Sep 25 15:18:21 dist-test-slave-p468 krb5kdc[1717](info): commencing operation
krb5kdc: starting...
W20260925 15:18:23.002746 12320 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349162370577-12320-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349162370577-12320-0/krb5kdc/test-user.keytab.
Sep 25 15:18:23 dist-test-slave-p468 krb5kdc[1717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349503, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:18:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:18:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.8.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab.
Entry for principal kudu/127.12.8.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab.
I20260925 15:18:23.257298 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349162370577-12320-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349162370577-12320-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349162370577-12320-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.62:37847
--webserver_interface=127.12.8.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab
--principal=kudu/127.12.8.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:36819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.62:37847
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:18:23.584220 1730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:18:23.584544 1730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:18:23.584705 1730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:18:23.591943 1730 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:18:23.592087 1730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:18:23.592185 1730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:18:23.592304 1730 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:18:23.592379 1730 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:18:23.601919 1730 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:36819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349162370577-12320-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349162370577-12320-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.8.62:37847
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab
--principal=kudu/127.12.8.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.62:37847
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.8.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349162370577-12320-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:18:23.603170 1730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:18:23.604476 1730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:18:23.616053 1735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:23.615983 1738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:23.616032 1736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:18:23.619655 1730 server_base.cc:1034] running on GCE node
I20260925 15:18:23.620505 1730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:18:23.621942 1730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:18:23.623270 1730 hybrid_clock.cc:648] HybridClock initialized: now 1790349503623150 us; error 45 us; skew 500 ppm
I20260925 15:18:23.623667 1730 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:18:23 dist-test-slave-p468 krb5kdc[1717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349503, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:18:23.632582 1730 init.cc:377] Logged in from keytab as kudu/127.12.8.62@KRBTEST.COM (short username kudu)
I20260925 15:18:23.634424 1730 webserver.cc:460] Webserver started at http://127.12.8.62:35461/ using document root <none> and password file <none>
I20260925 15:18:23.635203 1730 fs_manager.cc:359] Metadata directory not provided
I20260925 15:18:23.635349 1730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:18:23.635696 1730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:18:23.638820 1730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349162370577-12320-0/minicluster-data/master-0/data/instance:
uuid: "d2ef928a309448a9bc7f84b17435cdcc"
format_stamp: "Formatted at 2026-09-25 15:18:23 on dist-test-slave-p468"
server_key: "44c492719cc68d46bcea3fb691a932fc"
server_key_iv: "105dfb9d19e43c7be4973b7a68e143d0"
server_key_version: "encryptionkey@0"
I20260925 15:18:23.639606 1730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349162370577-12320-0/minicluster-data/master-0/wal/instance:
uuid: "d2ef928a309448a9bc7f84b17435cdcc"
format_stamp: "Formatted at 2026-09-25 15:18:23 on dist-test-slave-p468"
server_key: "44c492719cc68d46bcea3fb691a932fc"
server_key_iv: "105dfb9d19e43c7be4973b7a68e143d0"
server_key_version: "encryptionkey@0"
I20260925 15:18:23.644692 1730 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 15:18:23.648054 1745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:23.649341 1730 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 15:18:23.649510 1730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349162370577-12320-0/minicluster-data/master-0/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349162370577-12320-0/minicluster-data/master-0/wal
uuid: "d2ef928a309448a9bc7f84b17435cdcc"
format_stamp: "Formatted at 2026-09-25 15:18:23 on dist-test-slave-p468"
server_key: "44c492719cc68d46bcea3fb691a932fc"
server_key_iv: "105dfb9d19e43c7be4973b7a68e143d0"
server_key_version: "encryptionkey@0"
I20260925 15:18:23.649701 1730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349162370577-12320-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349162370577-12320-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349162370577-12320-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:18:23.655347 1730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:18:23.659200 1730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:18:23.659483 1730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:18:23.678143 1730 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.62:37847
I20260925 15:18:23.678265 1796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.62:37847 every 8 connection(s)
I20260925 15:18:23.679584 1730 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
I20260925 15:18:23.680994 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 1730
I20260925 15:18:23.681161 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349162370577-12320-0/minicluster-data/master-0/wal/instance
I20260925 15:18:23.681710 12320 external_mini_cluster.cc:1428] Setting key 6eeeb85bb6eca76c96c0159cbb8318d6
I20260925 15:18:23.684072 1797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:18:23.689409 1800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:23.683503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39319 (local address 127.12.8.62:37847)
0925 15:18:23.684301 (+ 798us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:23.684316 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:23.684361 (+ 45us) server_negotiation.cc:407] Connection header received
0925 15:18:23.685970 (+ 1609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:23.685990 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:23.686381 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:23.686759 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:23.687752 (+ 993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:23.687757 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:23.687769 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:18:23.687792 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:23.687861 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:18:23.687879 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:23.688133 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:23.688274 (+ 141us) server_negotiation.cc:299] Negotiation successful
0925 15:18:23.688455 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":310,"threads_started":1}
I20260925 15:18:23.692834 12320 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:18:23.695048 1797 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2ef928a309448a9bc7f84b17435cdcc: Bootstrap starting.
I20260925 15:18:23.698443 1797 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d2ef928a309448a9bc7f84b17435cdcc: Neither blocks nor log segments found. Creating new log.
I20260925 15:18:23.699390 1800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:23.696869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47907 (local address 127.12.8.62:37847)
0925 15:18:23.697003 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:23.697007 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:23.697471 (+ 464us) server_negotiation.cc:407] Connection header received
0925 15:18:23.697644 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:23.697649 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:23.697697 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:23.697790 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:23.698732 (+ 942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:23.698737 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:23.698742 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:18:23.698762 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:23.698811 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:18:23.698826 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:23.698934 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:23.699059 (+ 125us) server_negotiation.cc:299] Negotiation successful
0925 15:18:23.699114 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 15:18:23.699646 1797 log.cc:824] T 00000000000000000000000000000000 P d2ef928a309448a9bc7f84b17435cdcc: Log is configured to *not* fsync() on all Append() calls
I20260925 15:18:23.702320 1797 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2ef928a309448a9bc7f84b17435cdcc: No bootstrap required, opened a new log
I20260925 15:18:23.709576 1797 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2ef928a309448a9bc7f84b17435cdcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2ef928a309448a9bc7f84b17435cdcc" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 37847 } }
I20260925 15:18:23.709969 1797 raft_consensus.cc:374] T 00000000000000000000000000000000 P d2ef928a309448a9bc7f84b17435cdcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:18:23.710076 1797 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2ef928a309448a9bc7f84b17435cdcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2ef928a309448a9bc7f84b17435cdcc, State: Initialized, Role: FOLLOWER
I20260925 15:18:23.710745 1797 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2ef928a309448a9bc7f84b17435cdcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2ef928a309448a9bc7f84b17435cdcc" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 37847 } }
I20260925 15:18:23.711038 1797 raft_consensus.cc:388] T 00000000000000000000000000000000 P d2ef928a309448a9bc7f84b17435cdcc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:18:23.711159 1797 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2ef928a309448a9bc7f84b17435cdcc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:18:23.711279 1797 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2ef928a309448a9bc7f84b17435cdcc [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:18:23.713275 1797 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2ef928a309448a9bc7f84b17435cdcc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2ef928a309448a9bc7f84b17435cdcc" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 37847 } }
I20260925 15:18:23.713703 1797 leader_election.cc:302] T 00000000000000000000000000000000 P d2ef928a309448a9bc7f84b17435cdcc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2ef928a309448a9bc7f84b17435cdcc; no voters:
I20260925 15:18:23.714265 1797 leader_election.cc:288] T 00000000000000000000000000000000 P d2ef928a309448a9bc7f84b17435cdcc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:18:23.714411 1807 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2ef928a309448a9bc7f84b17435cdcc [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:18:23.715085 1807 raft_consensus.cc:686] T 00000000000000000000000000000000 P d2ef928a309448a9bc7f84b17435cdcc [term 1 LEADER]: Becoming Leader. State: Replica: d2ef928a309448a9bc7f84b17435cdcc, State: Running, Role: LEADER
I20260925 15:18:23.715638 1807 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2ef928a309448a9bc7f84b17435cdcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2ef928a309448a9bc7f84b17435cdcc" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 37847 } }
I20260925 15:18:23.716509 1797 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2ef928a309448a9bc7f84b17435cdcc [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:18:23.718958 1808 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2ef928a309448a9bc7f84b17435cdcc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d2ef928a309448a9bc7f84b17435cdcc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2ef928a309448a9bc7f84b17435cdcc" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 37847 } } }
I20260925 15:18:23.719416 1808 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2ef928a309448a9bc7f84b17435cdcc [sys.catalog]: This master's current role is: LEADER
I20260925 15:18:23.718931 1809 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2ef928a309448a9bc7f84b17435cdcc [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2ef928a309448a9bc7f84b17435cdcc. Latest consensus state: current_term: 1 leader_uuid: "d2ef928a309448a9bc7f84b17435cdcc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2ef928a309448a9bc7f84b17435cdcc" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 37847 } } }
I20260925 15:18:23.720207 1809 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2ef928a309448a9bc7f84b17435cdcc [sys.catalog]: This master's current role is: LEADER
I20260925 15:18:23.721179 1814 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:18:23.727255 1814 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:18:23.736385 1814 catalog_manager.cc:1347] Generated new cluster ID: 885d85be08384d61b79e805233eee021
I20260925 15:18:23.736519 1814 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:18:23.763523 1814 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:18:23.764026 1814 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:18:23.781769 1814 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d2ef928a309448a9bc7f84b17435cdcc: Generated new TSK 0
I20260925 15:18:23.782725 1814 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:18:23.803599 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 1730
W20260925 15:18:23.804569 1792 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:39319
W20260925 15:18:23.805228 12320 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 1730: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 15:18:23.805364 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 1730
2026-09-25T15:18:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1900 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349162370577-12320-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:18:23 dist-test-slave-p468 krb5kdc[1828](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:18:23 dist-test-slave-p468 krb5kdc[1828](info): set up 2 sockets
Sep 25 15:18:23 dist-test-slave-p468 krb5kdc[1828](info): commencing operation
krb5kdc: starting...
W20260925 15:18:24.906680 12320 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349162370577-12320-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349162370577-12320-0/krb5kdc/test-user.keytab.
Sep 25 15:18:25 dist-test-slave-p468 krb5kdc[1828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349505, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:18:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:18:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.8.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab.
Entry for principal kudu/127.12.8.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab.
I20260925 15:18:25.156999 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349162370577-12320-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349162370577-12320-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349162370577-12320-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.62:38173
--webserver_interface=127.12.8.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab
--principal=kudu/127.12.8.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:38173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.62:38173
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:18:25.485554 1841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:18:25.485837 1841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:18:25.486017 1841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:18:25.493412 1841 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:18:25.493534 1841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:18:25.493619 1841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:18:25.493717 1841 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:18:25.493791 1841 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:18:25.503602 1841 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:38173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349162370577-12320-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349162370577-12320-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.8.62:38173
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab
--principal=kudu/127.12.8.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.62:38173
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.8.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349162370577-12320-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:18:25.504864 1841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:18:25.506157 1841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:18:25.517756 1847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:25.517715 1849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:25.517794 1846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:18:25.518556 1841 server_base.cc:1034] running on GCE node
I20260925 15:18:25.519347 1841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:18:25.520709 1841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:18:25.521931 1841 hybrid_clock.cc:648] HybridClock initialized: now 1790349505521900 us; error 37 us; skew 500 ppm
I20260925 15:18:25.522338 1841 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:18:25 dist-test-slave-p468 krb5kdc[1828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349505, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:18:25.531306 1841 init.cc:377] Logged in from keytab as kudu/127.12.8.62@KRBTEST.COM (short username kudu)
I20260925 15:18:25.532778 1841 webserver.cc:460] Webserver started at http://127.12.8.62:33395/ using document root <none> and password file <none>
I20260925 15:18:25.533489 1841 fs_manager.cc:359] Metadata directory not provided
I20260925 15:18:25.533622 1841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:18:25.533917 1841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:18:25.536947 1841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349162370577-12320-0/minicluster-data/master-0/data/instance:
uuid: "ba49282b075b4581abf149ff1383d8ed"
format_stamp: "Formatted at 2026-09-25 15:18:25 on dist-test-slave-p468"
server_key: "c746700aec20606435b704acb7ec0c17"
server_key_iv: "0a5d3060f24a3f75b894a54ecddf14ba"
server_key_version: "encryptionkey@0"
I20260925 15:18:25.537690 1841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349162370577-12320-0/minicluster-data/master-0/wal/instance:
uuid: "ba49282b075b4581abf149ff1383d8ed"
format_stamp: "Formatted at 2026-09-25 15:18:25 on dist-test-slave-p468"
server_key: "c746700aec20606435b704acb7ec0c17"
server_key_iv: "0a5d3060f24a3f75b894a54ecddf14ba"
server_key_version: "encryptionkey@0"
I20260925 15:18:25.542964 1841 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260925 15:18:25.546358 1856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:25.547721 1841 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 15:18:25.547901 1841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349162370577-12320-0/minicluster-data/master-0/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349162370577-12320-0/minicluster-data/master-0/wal
uuid: "ba49282b075b4581abf149ff1383d8ed"
format_stamp: "Formatted at 2026-09-25 15:18:25 on dist-test-slave-p468"
server_key: "c746700aec20606435b704acb7ec0c17"
server_key_iv: "0a5d3060f24a3f75b894a54ecddf14ba"
server_key_version: "encryptionkey@0"
I20260925 15:18:25.548099 1841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349162370577-12320-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349162370577-12320-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349162370577-12320-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:18:25.560086 1841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:18:25.564040 1841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:18:25.564306 1841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:18:25.590912 1841 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.62:38173
I20260925 15:18:25.590926 1907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.62:38173 every 8 connection(s)
I20260925 15:18:25.592990 1841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
I20260925 15:18:25.597779 1908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:18:25.603134 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 1841
I20260925 15:18:25.603329 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349162370577-12320-0/minicluster-data/master-0/wal/instance
I20260925 15:18:25.603941 12320 external_mini_cluster.cc:1428] Setting key ed6c5a20c60a4a4e1f9d2e869dc6263d
I20260925 15:18:25.614821 1908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba49282b075b4581abf149ff1383d8ed: Bootstrap starting.
Sep 25 15:18:25 dist-test-slave-p468 krb5kdc[1828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349505, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
I20260925 15:18:25.619656 1908 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba49282b075b4581abf149ff1383d8ed: Neither blocks nor log segments found. Creating new log.
I20260925 15:18:25.621232 1908 log.cc:824] T 00000000000000000000000000000000 P ba49282b075b4581abf149ff1383d8ed: Log is configured to *not* fsync() on all Append() calls
I20260925 15:18:25.625121 1908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba49282b075b4581abf149ff1383d8ed: No bootstrap required, opened a new log
I20260925 15:18:25.628093 1911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:25.605972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51747 (local address 127.12.8.62:38173)
0925 15:18:25.606870 (+ 898us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:25.606881 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:25.606922 (+ 41us) server_negotiation.cc:407] Connection header received
0925 15:18:25.608612 (+ 1690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:25.608634 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:25.609018 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:25.609412 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:25.611104 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:25.613366 (+ 2262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:25.615462 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:25.616254 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:25.616436 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:25.620559 (+ 4123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:25.620566 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:25.620580 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:25.620613 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:25.624148 (+ 3535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:25.624780 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:25.624786 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:25.624791 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:25.624951 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:25.625351 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:25.625355 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:25.625357 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:25.625861 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:25.626177 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:25.626616 (+ 439us) server_negotiation.cc:299] Negotiation successful
0925 15:18:25.627071 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":666,"thread_start_us":433,"threads_started":1}
I20260925 15:18:25.631671 12320 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:18:25.634701 1908 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba49282b075b4581abf149ff1383d8ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba49282b075b4581abf149ff1383d8ed" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 38173 } }
I20260925 15:18:25.635102 1908 raft_consensus.cc:374] T 00000000000000000000000000000000 P ba49282b075b4581abf149ff1383d8ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:18:25.635223 1908 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba49282b075b4581abf149ff1383d8ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba49282b075b4581abf149ff1383d8ed, State: Initialized, Role: FOLLOWER
I20260925 15:18:25.635867 1908 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba49282b075b4581abf149ff1383d8ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba49282b075b4581abf149ff1383d8ed" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 38173 } }
I20260925 15:18:25.636206 1908 raft_consensus.cc:388] T 00000000000000000000000000000000 P ba49282b075b4581abf149ff1383d8ed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:18:25.636363 1908 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba49282b075b4581abf149ff1383d8ed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:18:25.636582 1908 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba49282b075b4581abf149ff1383d8ed [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:18:25.639032 1908 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba49282b075b4581abf149ff1383d8ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba49282b075b4581abf149ff1383d8ed" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 38173 } }
I20260925 15:18:25.639564 1908 leader_election.cc:302] T 00000000000000000000000000000000 P ba49282b075b4581abf149ff1383d8ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba49282b075b4581abf149ff1383d8ed; no voters:
I20260925 15:18:25.640139 1908 leader_election.cc:288] T 00000000000000000000000000000000 P ba49282b075b4581abf149ff1383d8ed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:18:25.640276 1918 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba49282b075b4581abf149ff1383d8ed [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:18:25.642503 1908 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba49282b075b4581abf149ff1383d8ed [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:18:25.647083 1918 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba49282b075b4581abf149ff1383d8ed [term 1 LEADER]: Becoming Leader. State: Replica: ba49282b075b4581abf149ff1383d8ed, State: Running, Role: LEADER
I20260925 15:18:25.647707 1918 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba49282b075b4581abf149ff1383d8ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba49282b075b4581abf149ff1383d8ed" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 38173 } }
I20260925 15:18:25.650828 1920 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba49282b075b4581abf149ff1383d8ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba49282b075b4581abf149ff1383d8ed. Latest consensus state: current_term: 1 leader_uuid: "ba49282b075b4581abf149ff1383d8ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba49282b075b4581abf149ff1383d8ed" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 38173 } } }
I20260925 15:18:25.651786 1920 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba49282b075b4581abf149ff1383d8ed [sys.catalog]: This master's current role is: LEADER
I20260925 15:18:25.653311 1919 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba49282b075b4581abf149ff1383d8ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ba49282b075b4581abf149ff1383d8ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba49282b075b4581abf149ff1383d8ed" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 38173 } } }
I20260925 15:18:25.658694 1919 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba49282b075b4581abf149ff1383d8ed [sys.catalog]: This master's current role is: LEADER
I20260925 15:18:25.661257 1911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:25.636082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60645 (local address 127.12.8.62:38173)
0925 15:18:25.636245 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:25.636249 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:25.636927 (+ 678us) server_negotiation.cc:407] Connection header received
0925 15:18:25.637115 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:25.637120 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:25.637202 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:25.637307 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:25.638681 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:25.639488 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:25.641503 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:25.642428 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:25.642604 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:25.646426 (+ 3822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:25.646436 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:25.646445 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:25.646512 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:25.651951 (+ 5439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:25.653020 (+ 1069us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:25.653025 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:25.653027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:25.653501 (+ 474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:25.654065 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:25.654074 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:25.654080 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:25.659269 (+ 5189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:25.659618 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:25.660366 (+ 748us) server_negotiation.cc:299] Negotiation successful
0925 15:18:25.660694 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":46}
I20260925 15:18:25.663633 1927 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:18:25.668510 1927 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:18:25.677675 1927 catalog_manager.cc:1347] Generated new cluster ID: 917962f8ed6f4100864a65a1992ee425
I20260925 15:18:25.677793 1927 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:18:25.688231 1927 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:18:25.690057 1927 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:18:25.696671 1927 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ba49282b075b4581abf149ff1383d8ed: Generated new TSK 0
I20260925 15:18:25.697608 1927 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:18:25.709748 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 1841
I20260925 15:18:25.711122 1903 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:18:25.799805 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 1841
2026-09-25T15:18:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1998 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349162370577-12320-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:18:25 dist-test-slave-p468 krb5kdc[1940](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:18:25 dist-test-slave-p468 krb5kdc[1940](info): set up 2 sockets
Sep 25 15:18:25 dist-test-slave-p468 krb5kdc[1940](info): commencing operation
krb5kdc: starting...
W20260925 15:18:26.902712 12320 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.004s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349162370577-12320-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349162370577-12320-0/krb5kdc/test-user.keytab.
Sep 25 15:18:27 dist-test-slave-p468 krb5kdc[1940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349507, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:18:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:18:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.8.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab.
Entry for principal kudu/127.12.8.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab.
I20260925 15:18:27.151523 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349162370577-12320-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349162370577-12320-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349162370577-12320-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.62:33483
--webserver_interface=127.12.8.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab
--principal=kudu/127.12.8.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:33809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.62:33483
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:18:27.481209 1953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:18:27.481565 1953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:18:27.481716 1953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:18:27.489406 1953 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:18:27.489560 1953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:18:27.489648 1953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:18:27.489742 1953 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:18:27.489809 1953 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:18:27.499763 1953 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:33809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349162370577-12320-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349162370577-12320-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.8.62:33483
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab
--principal=kudu/127.12.8.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.62:33483
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.8.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349162370577-12320-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:18:27.501050 1953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:18:27.502408 1953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:18:27.513985 1958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:27.513923 1959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:27.513964 1961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:18:27.514652 1953 server_base.cc:1034] running on GCE node
I20260925 15:18:27.515384 1953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:18:27.516680 1953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:18:27.517886 1953 hybrid_clock.cc:648] HybridClock initialized: now 1790349507517830 us; error 37 us; skew 500 ppm
I20260925 15:18:27.518270 1953 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:18:27 dist-test-slave-p468 krb5kdc[1940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349507, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:18:27.521371 1953 init.cc:377] Logged in from keytab as kudu/127.12.8.62@KRBTEST.COM (short username kudu)
I20260925 15:18:27.522892 1953 webserver.cc:460] Webserver started at http://127.12.8.62:45079/ using document root <none> and password file <none>
I20260925 15:18:27.523566 1953 fs_manager.cc:359] Metadata directory not provided
I20260925 15:18:27.523692 1953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:18:27.523998 1953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:18:27.526974 1953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349162370577-12320-0/minicluster-data/master-0/data/instance:
uuid: "74899abff143485b92ee982f0dd9b2c9"
format_stamp: "Formatted at 2026-09-25 15:18:27 on dist-test-slave-p468"
server_key: "43a4b7f8c1162da906862e0926a92551"
server_key_iv: "88ee6a1fc9050a889bfe0dbf113d9b2e"
server_key_version: "encryptionkey@0"
I20260925 15:18:27.527681 1953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349162370577-12320-0/minicluster-data/master-0/wal/instance:
uuid: "74899abff143485b92ee982f0dd9b2c9"
format_stamp: "Formatted at 2026-09-25 15:18:27 on dist-test-slave-p468"
server_key: "43a4b7f8c1162da906862e0926a92551"
server_key_iv: "88ee6a1fc9050a889bfe0dbf113d9b2e"
server_key_version: "encryptionkey@0"
I20260925 15:18:27.532562 1953 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 15:18:27.535859 1968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:27.537111 1953 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 15:18:27.537281 1953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349162370577-12320-0/minicluster-data/master-0/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349162370577-12320-0/minicluster-data/master-0/wal
uuid: "74899abff143485b92ee982f0dd9b2c9"
format_stamp: "Formatted at 2026-09-25 15:18:27 on dist-test-slave-p468"
server_key: "43a4b7f8c1162da906862e0926a92551"
server_key_iv: "88ee6a1fc9050a889bfe0dbf113d9b2e"
server_key_version: "encryptionkey@0"
I20260925 15:18:27.537457 1953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349162370577-12320-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349162370577-12320-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349162370577-12320-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:18:27.561180 1953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:18:27.565291 1953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:18:27.565600 1953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:18:27.583850 1953 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.62:33483
I20260925 15:18:27.583921 2019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.62:33483 every 8 connection(s)
I20260925 15:18:27.585194 1953 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
I20260925 15:18:27.588797 2020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:18:27.595099 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 1953
I20260925 15:18:27.595273 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349162370577-12320-0/minicluster-data/master-0/wal/instance
I20260925 15:18:27.595713 12320 external_mini_cluster.cc:1428] Setting key 698e9dd2eb3c07832cac04230c830f7b
I20260925 15:18:27.599372 2020 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74899abff143485b92ee982f0dd9b2c9: Bootstrap starting.
I20260925 15:18:27.602761 2020 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 74899abff143485b92ee982f0dd9b2c9: Neither blocks nor log segments found. Creating new log.
I20260925 15:18:27.604061 2020 log.cc:824] T 00000000000000000000000000000000 P 74899abff143485b92ee982f0dd9b2c9: Log is configured to *not* fsync() on all Append() calls
I20260925 15:18:27.607334 2020 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74899abff143485b92ee982f0dd9b2c9: No bootstrap required, opened a new log
I20260925 15:18:27.609447 2023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:27.597839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38085 (local address 127.12.8.62:33483)
0925 15:18:27.598664 (+ 825us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:27.598674 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:27.598712 (+ 38us) server_negotiation.cc:407] Connection header received
0925 15:18:27.600402 (+ 1690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:27.600422 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:27.600801 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:27.601135 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:27.601691 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:27.603055 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:27.604953 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:27.605801 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:27.605940 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:27.607211 (+ 1271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:27.607218 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:27.607231 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:18:27.607800 (+ 569us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:27.607877 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:18:27.607904 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:27.607990 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:27.608218 (+ 228us) server_negotiation.cc:299] Negotiation successful
0925 15:18:27.608482 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":621,"thread_start_us":429,"threads_started":1}
I20260925 15:18:27.613013 12320 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:18:27.615067 2020 raft_consensus.cc:348] T 00000000000000000000000000000000 P 74899abff143485b92ee982f0dd9b2c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74899abff143485b92ee982f0dd9b2c9" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 33483 } }
I20260925 15:18:27.615443 2020 raft_consensus.cc:374] T 00000000000000000000000000000000 P 74899abff143485b92ee982f0dd9b2c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:18:27.615545 2020 raft_consensus.cc:729] T 00000000000000000000000000000000 P 74899abff143485b92ee982f0dd9b2c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74899abff143485b92ee982f0dd9b2c9, State: Initialized, Role: FOLLOWER
I20260925 15:18:27.616175 2020 consensus_queue.cc:260] T 00000000000000000000000000000000 P 74899abff143485b92ee982f0dd9b2c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74899abff143485b92ee982f0dd9b2c9" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 33483 } }
I20260925 15:18:27.616441 2020 raft_consensus.cc:388] T 00000000000000000000000000000000 P 74899abff143485b92ee982f0dd9b2c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:18:27.616588 2020 raft_consensus.cc:482] T 00000000000000000000000000000000 P 74899abff143485b92ee982f0dd9b2c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:18:27.616760 2020 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 74899abff143485b92ee982f0dd9b2c9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:18:27.619777 2020 raft_consensus.cc:504] T 00000000000000000000000000000000 P 74899abff143485b92ee982f0dd9b2c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74899abff143485b92ee982f0dd9b2c9" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 33483 } }
I20260925 15:18:27.620273 2020 leader_election.cc:302] T 00000000000000000000000000000000 P 74899abff143485b92ee982f0dd9b2c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 74899abff143485b92ee982f0dd9b2c9; no voters:
I20260925 15:18:27.620894 2020 leader_election.cc:288] T 00000000000000000000000000000000 P 74899abff143485b92ee982f0dd9b2c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:18:27.621028 2030 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 74899abff143485b92ee982f0dd9b2c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:18:27.621649 2030 raft_consensus.cc:686] T 00000000000000000000000000000000 P 74899abff143485b92ee982f0dd9b2c9 [term 1 LEADER]: Becoming Leader. State: Replica: 74899abff143485b92ee982f0dd9b2c9, State: Running, Role: LEADER
I20260925 15:18:27.622262 2030 consensus_queue.cc:237] T 00000000000000000000000000000000 P 74899abff143485b92ee982f0dd9b2c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74899abff143485b92ee982f0dd9b2c9" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 33483 } }
I20260925 15:18:27.624296 2030 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74899abff143485b92ee982f0dd9b2c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 74899abff143485b92ee982f0dd9b2c9. Latest consensus state: current_term: 1 leader_uuid: "74899abff143485b92ee982f0dd9b2c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74899abff143485b92ee982f0dd9b2c9" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 33483 } } }
I20260925 15:18:27.624598 2030 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74899abff143485b92ee982f0dd9b2c9 [sys.catalog]: This master's current role is: LEADER
I20260925 15:18:27.626067 2023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:27.617296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34023 (local address 127.12.8.62:33483)
0925 15:18:27.617478 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:27.617484 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:27.618018 (+ 534us) server_negotiation.cc:407] Connection header received
0925 15:18:27.618140 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:27.618145 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:27.618263 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:27.618363 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:27.618822 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:27.619851 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:27.621582 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:27.622346 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:27.622493 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:27.624283 (+ 1790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:27.624293 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:27.624301 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:18:27.624343 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:27.624408 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:18:27.624436 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:27.625374 (+ 938us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:27.625446 (+ 72us) server_negotiation.cc:299] Negotiation successful
0925 15:18:27.625559 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 15:18:27.625913 2032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74899abff143485b92ee982f0dd9b2c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "74899abff143485b92ee982f0dd9b2c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74899abff143485b92ee982f0dd9b2c9" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 33483 } } }
I20260925 15:18:27.626655 2032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74899abff143485b92ee982f0dd9b2c9 [sys.catalog]: This master's current role is: LEADER
I20260925 15:18:27.628949 2020 sys_catalog.cc:564] T 00000000000000000000000000000000 P 74899abff143485b92ee982f0dd9b2c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:18:27.629187 2034 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:18:27.634661 2034 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:18:27.645609 2034 catalog_manager.cc:1347] Generated new cluster ID: 608490cc34b344a8a8b815913668c563
I20260925 15:18:27.645714 2034 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:18:27.676374 2034 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:18:27.677762 2034 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:18:27.685246 2034 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 74899abff143485b92ee982f0dd9b2c9: Generated new TSK 0
I20260925 15:18:27.686246 2034 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:18:27.701459 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 1953
W20260925 15:18:27.702538 2015 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:38085
W20260925 15:18:27.703212 12320 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 1953: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 15:18:27.703354 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 1953
2026-09-25T15:18:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1903 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349162370577-12320-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:18:27 dist-test-slave-p468 krb5kdc[2051](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:18:27 dist-test-slave-p468 krb5kdc[2051](info): set up 2 sockets
Sep 25 15:18:27 dist-test-slave-p468 krb5kdc[2051](info): commencing operation
krb5kdc: starting...
W20260925 15:18:28.830770 12320 mini_kdc.cc:121] Time spent starting KDC: real 1.092s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349162370577-12320-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349162370577-12320-0/krb5kdc/test-user.keytab.
Sep 25 15:18:28 dist-test-slave-p468 krb5kdc[2051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349508, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:18:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:18:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.8.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab.
Entry for principal kudu/127.12.8.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab.
I20260925 15:18:29.087007 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349162370577-12320-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349162370577-12320-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349162370577-12320-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.62:37059
--webserver_interface=127.12.8.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab
--principal=kudu/127.12.8.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:42399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.62:37059
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:18:29.412034 2064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:18:29.412312 2064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:18:29.412487 2064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:18:29.419556 2064 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:18:29.419657 2064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:18:29.419760 2064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:18:29.419866 2064 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:18:29.419948 2064 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:18:29.429664 2064 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:42399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349162370577-12320-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349162370577-12320-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.8.62:37059
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab
--principal=kudu/127.12.8.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.62:37059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.8.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349162370577-12320-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:18:29.430956 2064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:18:29.432286 2064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:18:29.443614 2072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:29.443650 2070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:29.443697 2069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:18:29.444712 2064 server_base.cc:1034] running on GCE node
I20260925 15:18:29.445441 2064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:18:29.446837 2064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:18:29.448087 2064 hybrid_clock.cc:648] HybridClock initialized: now 1790349509448043 us; error 57 us; skew 500 ppm
I20260925 15:18:29.448429 2064 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:18:29 dist-test-slave-p468 krb5kdc[2051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349509, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:18:29.461977 2064 init.cc:377] Logged in from keytab as kudu/127.12.8.62@KRBTEST.COM (short username kudu)
I20260925 15:18:29.463678 2064 webserver.cc:460] Webserver started at http://127.12.8.62:38479/ using document root <none> and password file <none>
I20260925 15:18:29.464388 2064 fs_manager.cc:359] Metadata directory not provided
I20260925 15:18:29.464538 2064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:18:29.464845 2064 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:18:29.467903 2064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349162370577-12320-0/minicluster-data/master-0/data/instance:
uuid: "6bd08d4c39e64829850e0eb4d1248c82"
format_stamp: "Formatted at 2026-09-25 15:18:29 on dist-test-slave-p468"
server_key: "e2101fd2976cda43ddb19d53f235b85c"
server_key_iv: "5c91983d48bf10e26859ad2ccd94807c"
server_key_version: "encryptionkey@0"
I20260925 15:18:29.468684 2064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349162370577-12320-0/minicluster-data/master-0/wal/instance:
uuid: "6bd08d4c39e64829850e0eb4d1248c82"
format_stamp: "Formatted at 2026-09-25 15:18:29 on dist-test-slave-p468"
server_key: "e2101fd2976cda43ddb19d53f235b85c"
server_key_iv: "5c91983d48bf10e26859ad2ccd94807c"
server_key_version: "encryptionkey@0"
I20260925 15:18:29.473735 2064 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 15:18:29.477291 2079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:29.478838 2064 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:18:29.479013 2064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349162370577-12320-0/minicluster-data/master-0/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349162370577-12320-0/minicluster-data/master-0/wal
uuid: "6bd08d4c39e64829850e0eb4d1248c82"
format_stamp: "Formatted at 2026-09-25 15:18:29 on dist-test-slave-p468"
server_key: "e2101fd2976cda43ddb19d53f235b85c"
server_key_iv: "5c91983d48bf10e26859ad2ccd94807c"
server_key_version: "encryptionkey@0"
I20260925 15:18:29.479209 2064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349162370577-12320-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349162370577-12320-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349162370577-12320-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:18:29.484936 2064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:18:29.489289 2064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:18:29.489593 2064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:18:29.507851 2064 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.62:37059
I20260925 15:18:29.507893 2130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.62:37059 every 8 connection(s)
I20260925 15:18:29.509341 2064 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
I20260925 15:18:29.510437 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 2064
I20260925 15:18:29.510637 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349162370577-12320-0/minicluster-data/master-0/wal/instance
I20260925 15:18:29.511148 12320 external_mini_cluster.cc:1428] Setting key c83a35f8bd46f069f79bb779d81f9276
I20260925 15:18:29.513993 2131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:18:29.518748 2134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:29.512947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54517 (local address 127.12.8.62:37059)
0925 15:18:29.513705 (+ 758us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:29.513716 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:29.513758 (+ 42us) server_negotiation.cc:407] Connection header received
0925 15:18:29.515452 (+ 1694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:29.515474 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:29.515787 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:29.516181 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:29.517192 (+ 1011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:29.517198 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:29.517209 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:18:29.517222 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:29.517288 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:18:29.517313 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:29.517542 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:29.517676 (+ 134us) server_negotiation.cc:299] Negotiation successful
0925 15:18:29.517837 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":309,"threads_started":1}
I20260925 15:18:29.522183 12320 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:18:29.524919 2131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bd08d4c39e64829850e0eb4d1248c82: Bootstrap starting.
I20260925 15:18:29.528429 2131 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6bd08d4c39e64829850e0eb4d1248c82: Neither blocks nor log segments found. Creating new log.
I20260925 15:18:29.528865 2134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:29.526477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44699 (local address 127.12.8.62:37059)
0925 15:18:29.526678 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:29.526683 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:29.526972 (+ 289us) server_negotiation.cc:407] Connection header received
0925 15:18:29.527140 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:29.527143 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:29.527185 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:29.527252 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:29.528181 (+ 929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:29.528184 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:29.528187 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:18:29.528200 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:29.528235 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:18:29.528249 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:29.528335 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:29.528481 (+ 146us) server_negotiation.cc:299] Negotiation successful
0925 15:18:29.528560 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260925 15:18:29.529603 2131 log.cc:824] T 00000000000000000000000000000000 P 6bd08d4c39e64829850e0eb4d1248c82: Log is configured to *not* fsync() on all Append() calls
I20260925 15:18:29.532122 2131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bd08d4c39e64829850e0eb4d1248c82: No bootstrap required, opened a new log
I20260925 15:18:29.539070 2131 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6bd08d4c39e64829850e0eb4d1248c82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bd08d4c39e64829850e0eb4d1248c82" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 37059 } }
I20260925 15:18:29.539445 2131 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6bd08d4c39e64829850e0eb4d1248c82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:18:29.539548 2131 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6bd08d4c39e64829850e0eb4d1248c82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bd08d4c39e64829850e0eb4d1248c82, State: Initialized, Role: FOLLOWER
I20260925 15:18:29.540184 2131 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6bd08d4c39e64829850e0eb4d1248c82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bd08d4c39e64829850e0eb4d1248c82" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 37059 } }
I20260925 15:18:29.540392 2131 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6bd08d4c39e64829850e0eb4d1248c82 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:18:29.540489 2131 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6bd08d4c39e64829850e0eb4d1248c82 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:18:29.540606 2131 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6bd08d4c39e64829850e0eb4d1248c82 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:18:29.542681 2131 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6bd08d4c39e64829850e0eb4d1248c82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bd08d4c39e64829850e0eb4d1248c82" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 37059 } }
I20260925 15:18:29.543099 2131 leader_election.cc:302] T 00000000000000000000000000000000 P 6bd08d4c39e64829850e0eb4d1248c82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6bd08d4c39e64829850e0eb4d1248c82; no voters:
I20260925 15:18:29.543568 2131 leader_election.cc:288] T 00000000000000000000000000000000 P 6bd08d4c39e64829850e0eb4d1248c82 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:18:29.543700 2141 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6bd08d4c39e64829850e0eb4d1248c82 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:18:29.544356 2141 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6bd08d4c39e64829850e0eb4d1248c82 [term 1 LEADER]: Becoming Leader. State: Replica: 6bd08d4c39e64829850e0eb4d1248c82, State: Running, Role: LEADER
I20260925 15:18:29.544914 2141 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6bd08d4c39e64829850e0eb4d1248c82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bd08d4c39e64829850e0eb4d1248c82" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 37059 } }
I20260925 15:18:29.545533 2131 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6bd08d4c39e64829850e0eb4d1248c82 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:18:29.547659 2143 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bd08d4c39e64829850e0eb4d1248c82 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6bd08d4c39e64829850e0eb4d1248c82. Latest consensus state: current_term: 1 leader_uuid: "6bd08d4c39e64829850e0eb4d1248c82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bd08d4c39e64829850e0eb4d1248c82" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 37059 } } }
I20260925 15:18:29.548019 2143 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bd08d4c39e64829850e0eb4d1248c82 [sys.catalog]: This master's current role is: LEADER
I20260925 15:18:29.549046 2142 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bd08d4c39e64829850e0eb4d1248c82 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6bd08d4c39e64829850e0eb4d1248c82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bd08d4c39e64829850e0eb4d1248c82" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 37059 } } }
I20260925 15:18:29.549490 2142 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bd08d4c39e64829850e0eb4d1248c82 [sys.catalog]: This master's current role is: LEADER
I20260925 15:18:29.550544 2147 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:18:29.555871 2147 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:18:29.564950 2147 catalog_manager.cc:1347] Generated new cluster ID: 6ed6416e630a452586f0625b6f098bd6
I20260925 15:18:29.565074 2147 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:18:29.581393 2147 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:18:29.581826 2147 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:18:29.592401 2147 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6bd08d4c39e64829850e0eb4d1248c82: Generated new TSK 0
I20260925 15:18:29.593389 2147 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:18:29.603341 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 2064
W20260925 15:18:29.604460 2126 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:54517
W20260925 15:18:29.605242 12320 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 2064: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 15:18:29.605489 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 2064
2026-09-25T15:18:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1901 ms)
[----------] 18 tests from AuthTokenIssuingTest (34790 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349162370577-12320-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:18:29 dist-test-slave-p468 krb5kdc[2162](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:18:29 dist-test-slave-p468 krb5kdc[2162](info): set up 2 sockets
Sep 25 15:18:29 dist-test-slave-p468 krb5kdc[2162](info): commencing operation
krb5kdc: starting...
W20260925 15:18:30.831254 12320 mini_kdc.cc:121] Time spent starting KDC: real 1.190s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349162370577-12320-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349162370577-12320-0/krb5kdc/test-user.keytab.
Sep 25 15:18:30 dist-test-slave-p468 krb5kdc[2162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:18:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:18:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.8.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab.
Entry for principal kudu/127.12.8.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab.
I20260925 15:18:31.087323 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349162370577-12320-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349162370577-12320-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349162370577-12320-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.62:44539
--webserver_interface=127.12.8.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab
--principal=kudu/127.12.8.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:46233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.62:44539,127.12.8.61:36113,127.12.8.60:46547
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:18:31.408422 2175 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 15:18:31.408874 2175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:18:31.409005 2175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:18:31.409152 2175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:18:31.416193 2175 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:18:31.416312 2175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:18:31.416411 2175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:18:31.416505 2175 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:18:31.416594 2175 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:18:31.426537 2175 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:46233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349162370577-12320-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349162370577-12320-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.8.62:44539,127.12.8.61:36113,127.12.8.60:46547
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab
--principal=kudu/127.12.8.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.62:44539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.8.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349162370577-12320-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:18:31.427794 2175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:18:31.429148 2175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:18:31.440696 2183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:31.440771 2180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:31.440768 2181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:18:31.441924 2175 server_base.cc:1034] running on GCE node
I20260925 15:18:31.442706 2175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:18:31.444005 2175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:18:31.445216 2175 hybrid_clock.cc:648] HybridClock initialized: now 1790349511445147 us; error 45 us; skew 500 ppm
I20260925 15:18:31.445528 2175 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:18:31 dist-test-slave-p468 krb5kdc[2162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349511, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:18:31.448554 2175 init.cc:377] Logged in from keytab as kudu/127.12.8.62@KRBTEST.COM (short username kudu)
I20260925 15:18:31.449857 2175 webserver.cc:460] Webserver started at http://127.12.8.62:41847/ using document root <none> and password file <none>
I20260925 15:18:31.450523 2175 fs_manager.cc:359] Metadata directory not provided
I20260925 15:18:31.450608 2175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:18:31.450874 2175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:18:31.453603 2175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349162370577-12320-0/minicluster-data/master-0/data/instance:
uuid: "a93fde3803b04be9b5f51407e2102ac5"
format_stamp: "Formatted at 2026-09-25 15:18:31 on dist-test-slave-p468"
server_key: "9cbce26b7d9ea2c1bb039fed96552e6d"
server_key_iv: "865be2cf060f94bf56d409dfe75b4ede"
server_key_version: "encryptionkey@0"
I20260925 15:18:31.454389 2175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349162370577-12320-0/minicluster-data/master-0/wal/instance:
uuid: "a93fde3803b04be9b5f51407e2102ac5"
format_stamp: "Formatted at 2026-09-25 15:18:31 on dist-test-slave-p468"
server_key: "9cbce26b7d9ea2c1bb039fed96552e6d"
server_key_iv: "865be2cf060f94bf56d409dfe75b4ede"
server_key_version: "encryptionkey@0"
I20260925 15:18:31.459193 2175 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 15:18:31.462483 2190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:31.463797 2175 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 15:18:31.463953 2175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349162370577-12320-0/minicluster-data/master-0/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349162370577-12320-0/minicluster-data/master-0/wal
uuid: "a93fde3803b04be9b5f51407e2102ac5"
format_stamp: "Formatted at 2026-09-25 15:18:31 on dist-test-slave-p468"
server_key: "9cbce26b7d9ea2c1bb039fed96552e6d"
server_key_iv: "865be2cf060f94bf56d409dfe75b4ede"
server_key_version: "encryptionkey@0"
I20260925 15:18:31.464157 2175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349162370577-12320-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349162370577-12320-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349162370577-12320-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:18:31.476186 2175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:18:31.480101 2175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:18:31.480366 2175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:18:31.499218 2175 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.62:44539
I20260925 15:18:31.499238 2241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.62:44539 every 8 connection(s)
I20260925 15:18:31.500610 2175 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
I20260925 15:18:31.504381 2242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:18:31.510596 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 2175
I20260925 15:18:31.510814 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349162370577-12320-0/minicluster-data/master-0/wal/instance
I20260925 15:18:31.511613 12320 external_mini_cluster.cc:1428] Setting key b696c84157b488eb9129b5c7bc7f0447
I20260925 15:18:31.511091 2242 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.8.62" port: 44539 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:18:31 dist-test-slave-p468 krb5kdc[2162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349511, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.62@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
I20260925 15:18:31.535018 2245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:31.514796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60846 (local address 127.12.8.62:44539)
0925 15:18:31.516671 (+ 1875us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:31.516677 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:31.516720 (+ 43us) server_negotiation.cc:407] Connection header received
0925 15:18:31.518878 (+ 2158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:31.518883 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:31.519100 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:31.519257 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:31.520887 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:31.522056 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:31.523429 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:31.523978 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:31.524183 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:31.527978 (+ 3795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:31.527986 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:31.527997 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:31.528025 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:31.530750 (+ 2725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:31.531635 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:31.531641 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:31.531643 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:31.531779 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:31.532274 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:31.532277 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:31.532279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:31.532768 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:31.532918 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:31.533656 (+ 738us) server_negotiation.cc:299] Negotiation successful
0925 15:18:31.533923 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":1692,"spinlock_wait_cycles":16768,"thread_start_us":1461,"threads_started":1}
I20260925 15:18:31.535109 2244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:31.514796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:44539 (local address 127.0.0.1:60846)
0925 15:18:31.515693 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:31.515740 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:31.518513 (+ 2773us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:31.519348 (+ 835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:31.519357 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:31.520167 (+ 810us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:18:31.520669 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:31.520694 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:31.522232 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:31.522240 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:31.523290 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:31.523301 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:31.524220 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:31.524226 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:31.524404 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:31.525062 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:18:31.525083 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:18:31.527766 (+ 2683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:18:31.530877 (+ 3111us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:31.530891 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:31.530905 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:31.531493 (+ 588us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:31.531892 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:31.531900 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:31.531907 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:31.532163 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:31.532913 (+ 750us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:18:31.532923 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:18:31.533223 (+ 300us) client_negotiation.cc:769] Sending connection context
0925 15:18:31.533510 (+ 287us) client_negotiation.cc:240] Negotiation successful
0925 15:18:31.534115 (+ 605us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"mutex_wait_us":69,"thread_start_us":356,"threads_started":1}
WARNING: no policy specified for kudu/127.12.8.61@KRBTEST.COM; defaulting to no policy
I20260925 15:18:31.537721 2242 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.8.61" port: 36113 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:18:31.538806 2244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:31.538446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.61:36113 (local address 127.0.0.1:60378)
0925 15:18:31.538611 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:31.538708 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.61:36113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260925 15:18:31.539067 2193 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.8.61:36113: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.61@KRBTEST.COM" created.
I20260925 15:18:31.540645 2244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:31.540334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.61:36113 (local address 127.0.0.1:60388)
0925 15:18:31.540474 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:31.540533 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.61:36113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260925 15:18:31.540881 2242 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.8.61:36113: Network error: Client connection negotiation failed: client connection to 127.12.8.61:36113: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.61.keytab.
Entry for principal kudu/127.12.8.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.61.keytab.
I20260925 15:18:31.569448 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349162370577-12320-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349162370577-12320-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349162370577-12320-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349162370577-12320-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.61:36113
--webserver_interface=127.12.8.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.61.keytab
--principal=kudu/127.12.8.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:46233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.62:44539,127.12.8.61:36113,127.12.8.60:46547
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:18:31.590294 2242 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.8.61" port: 36113 } attempt: 1
I20260925 15:18:31.591536 2244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:31.590943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.61:36113 (local address 127.0.0.1:60398)
0925 15:18:31.591198 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:31.591325 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.61:36113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260925 15:18:31.592732 2244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:31.592243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.61:36113 (local address 127.0.0.1:60410)
0925 15:18:31.592447 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:31.592545 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.61:36113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260925 15:18:31.592917 2242 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.8.61:36113: Network error: Client connection negotiation failed: client connection to 127.12.8.61:36113: connect: Connection refused (error 111)
I20260925 15:18:31.661183 2242 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.8.61" port: 36113 } attempt: 2
I20260925 15:18:31.662542 2244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:31.661944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.61:36113 (local address 127.0.0.1:60414)
0925 15:18:31.662244 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:31.662353 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.61:36113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260925 15:18:31.663852 2244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:31.663287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.61:36113 (local address 127.0.0.1:60418)
0925 15:18:31.663500 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:31.663637 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.61:36113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260925 15:18:31.664115 2242 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.8.61:36113: Network error: Client connection negotiation failed: client connection to 127.12.8.61:36113: connect: Connection refused (error 111)
I20260925 15:18:31.755407 2242 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.8.61" port: 36113 } attempt: 3
I20260925 15:18:31.756836 2244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:31.756209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.61:36113 (local address 127.0.0.1:60430)
0925 15:18:31.756485 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:31.756618 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.61:36113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260925 15:18:31.758162 2244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:31.757631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.61:36113 (local address 127.0.0.1:60434)
0925 15:18:31.757816 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:31.757933 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.61:36113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 15:18:31.758442 2242 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.8.61:36113: Network error: Client connection negotiation failed: client connection to 127.12.8.61:36113: connect: Connection refused (error 111)
W20260925 15:18:31.899207 2249 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 15:18:31.899614 2249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:18:31.899745 2249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:18:31.899886 2249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260925 15:18:31.901757 2242 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.8.61" port: 36113 } attempt: 4
I20260925 15:18:31.903122 2244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:31.902459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.61:36113 (local address 127.0.0.1:60438)
0925 15:18:31.902837 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:31.902940 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.61:36113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210}
I20260925 15:18:31.904417 2244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:31.903993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.61:36113 (local address 127.0.0.1:60450)
0925 15:18:31.904162 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:31.904266 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.61:36113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260925 15:18:31.904654 2242 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.8.61:36113: Network error: Client connection negotiation failed: client connection to 127.12.8.61:36113: connect: Connection refused (error 111)
W20260925 15:18:31.908089 2249 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:18:31.908172 2249 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:18:31.908228 2249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:18:31.908278 2249 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:18:31.908334 2249 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:18:31.917997 2249 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:46233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349162370577-12320-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349162370577-12320-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.12.8.62:44539,127.12.8.61:36113,127.12.8.60:46547
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.61.keytab
--principal=kudu/127.12.8.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.61:36113
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349162370577-12320-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.12.8.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349162370577-12320-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:18:31.919283 2249 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:18:31.920555 2249 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:18:31.932094 2257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:31.932154 2254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:31.932358 2255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:18:31.933455 2249 server_base.cc:1034] running on GCE node
I20260925 15:18:31.934288 2249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:18:31.935634 2249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:18:31.936861 2249 hybrid_clock.cc:648] HybridClock initialized: now 1790349511936776 us; error 56 us; skew 500 ppm
I20260925 15:18:31.937165 2249 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:18:31 dist-test-slave-p468 krb5kdc[2162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349511, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:18:31.940377 2249 init.cc:377] Logged in from keytab as kudu/127.12.8.61@KRBTEST.COM (short username kudu)
I20260925 15:18:31.941847 2249 webserver.cc:460] Webserver started at http://127.12.8.61:32781/ using document root <none> and password file <none>
I20260925 15:18:31.942574 2249 fs_manager.cc:359] Metadata directory not provided
I20260925 15:18:31.942662 2249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:18:31.942914 2249 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:18:31.945844 2249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349162370577-12320-0/minicluster-data/master-1/data/instance:
uuid: "a7db4ff52dca4ab08b2a471c0c94df52"
format_stamp: "Formatted at 2026-09-25 15:18:31 on dist-test-slave-p468"
server_key: "bc74e1f23f29dd45ca8c5148ada01761"
server_key_iv: "f112ac768316abac223f82da2059512d"
server_key_version: "encryptionkey@0"
I20260925 15:18:31.946702 2249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349162370577-12320-0/minicluster-data/master-1/wal/instance:
uuid: "a7db4ff52dca4ab08b2a471c0c94df52"
format_stamp: "Formatted at 2026-09-25 15:18:31 on dist-test-slave-p468"
server_key: "bc74e1f23f29dd45ca8c5148ada01761"
server_key_iv: "f112ac768316abac223f82da2059512d"
server_key_version: "encryptionkey@0"
I20260925 15:18:31.952075 2249 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 15:18:31.955565 2264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:31.956928 2249 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 15:18:31.957134 2249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349162370577-12320-0/minicluster-data/master-1/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349162370577-12320-0/minicluster-data/master-1/wal
uuid: "a7db4ff52dca4ab08b2a471c0c94df52"
format_stamp: "Formatted at 2026-09-25 15:18:31 on dist-test-slave-p468"
server_key: "bc74e1f23f29dd45ca8c5148ada01761"
server_key_iv: "f112ac768316abac223f82da2059512d"
server_key_version: "encryptionkey@0"
I20260925 15:18:31.957362 2249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349162370577-12320-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349162370577-12320-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349162370577-12320-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:18:31.973186 2249 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:18:31.977811 2249 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:18:31.978113 2249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:18:31.997912 2249 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.61:36113
I20260925 15:18:31.997931 2315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.61:36113 every 8 connection(s)
I20260925 15:18:31.999387 2249 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349162370577-12320-0/minicluster-data/master-1/data/info.pb
I20260925 15:18:32.003139 2316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:18:32.006368 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 2249
I20260925 15:18:32.006553 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349162370577-12320-0/minicluster-data/master-1/wal/instance
I20260925 15:18:32.007097 12320 external_mini_cluster.cc:1428] Setting key 965ecbd81503f76fe0a67b62878a3d4b
I20260925 15:18:32.011688 2316 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.8.62" port: 44539 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:18:32 dist-test-slave-p468 krb5kdc[2162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349511, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.61@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.8.60@KRBTEST.COM; defaulting to no policy
I20260925 15:18:32.034554 2245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:32.015055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60854 (local address 127.12.8.62:44539)
0925 15:18:32.015299 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:32.015308 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:32.016494 (+ 1186us) server_negotiation.cc:407] Connection header received
0925 15:18:32.018462 (+ 1968us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:32.018470 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:32.018566 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:32.018691 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:32.020176 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:32.021319 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:32.022929 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:32.023682 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:32.023831 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:32.027257 (+ 3426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:32.027266 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:32.027274 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:32.027321 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:32.031287 (+ 3966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:32.032042 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:32.032050 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:32.032054 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:32.032224 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:32.032650 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:32.032658 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:32.032663 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:32.033057 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:32.033255 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:32.034020 (+ 765us) server_negotiation.cc:299] Negotiation successful
0925 15:18:32.034320 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260925 15:18:32.035594 2318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:32.015505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:44539 (local address 127.0.0.1:60854)
0925 15:18:32.016312 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:32.016355 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:32.018152 (+ 1797us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:32.018905 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:32.018913 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:32.019414 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:18:32.019985 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:32.020001 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:32.021478 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:32.021482 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:32.022786 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:32.022799 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:32.023880 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:32.023884 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:32.024071 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:32.024773 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:18:32.024796 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:18:32.027012 (+ 2216us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:18:32.031438 (+ 4426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:32.031446 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:32.031460 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:32.031900 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:32.032349 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:32.032352 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:32.032355 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:32.032524 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:32.033205 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:18:32.033211 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:18:32.033551 (+ 340us) client_negotiation.cc:769] Sending connection context
0925 15:18:32.033822 (+ 271us) client_negotiation.cc:240] Negotiation successful
0925 15:18:32.034361 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":367,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.60@KRBTEST.COM" created.
I20260925 15:18:32.037328 2316 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.8.61" port: 36113 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:18:32 dist-test-slave-p468 krb5kdc[2162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349511, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.61@KRBTEST.COM for kudu/127.12.8.61@KRBTEST.COM
I20260925 15:18:32.054626 2318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:32.038139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.61:36113 (local address 127.0.0.1:60466)
0925 15:18:32.038497 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:32.038517 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:32.038623 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:32.039431 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:32.039442 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:32.039945 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:18:32.040150 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:32.040174 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:32.042015 (+ 1841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:32.042024 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:32.043530 (+ 1506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:32.043547 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:32.044661 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:32.044669 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:32.044900 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:32.045511 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:18:32.045537 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:18:32.048063 (+ 2526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:18:32.052323 (+ 4260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:32.052327 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:32.052329 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:32.052693 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:32.053092 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:32.053095 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:32.053097 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:32.053209 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:32.053918 (+ 709us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:18:32.053921 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:18:32.054029 (+ 108us) client_negotiation.cc:769] Sending connection context
0925 15:18:32.054129 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 15:18:32.054379 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203}
I20260925 15:18:32.055117 2320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:32.038329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60466 (local address 127.12.8.61:36113)
0925 15:18:32.039020 (+ 691us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:32.039026 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:32.039050 (+ 24us) server_negotiation.cc:407] Connection header received
0925 15:18:32.039093 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:32.039098 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:32.039272 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:32.039388 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:32.040336 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:32.041846 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:32.043670 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:32.044479 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:32.044645 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:32.048286 (+ 3641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:32.048302 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:32.048322 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:32.048363 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:32.052205 (+ 3842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:32.052834 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:32.052845 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:32.052851 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:32.052987 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:32.053326 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:32.053334 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:32.053340 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:32.053789 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:32.053997 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:32.054347 (+ 350us) server_negotiation.cc:299] Negotiation successful
0925 15:18:32.054637 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":407,"threads_started":1}
I20260925 15:18:32.057598 2316 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.8.60" port: 46547 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:18:32.058776 2318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:32.058340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.60:46547 (local address 127.0.0.1:48998)
0925 15:18:32.058500 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:32.058625 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.60:46547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260925 15:18:32.059056 2268 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.8.60:46547: connect: Connection refused (error 111)
I20260925 15:18:32.060801 2318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:32.060318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.60:46547 (local address 127.0.0.1:49010)
0925 15:18:32.060554 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:32.060674 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.60:46547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
W20260925 15:18:32.061038 2316 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.8.60:46547: Network error: Client connection negotiation failed: client connection to 127.12.8.60:46547: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.60.keytab.
Entry for principal kudu/127.12.8.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.60.keytab.
I20260925 15:18:32.071074 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349162370577-12320-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349162370577-12320-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349162370577-12320-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349162370577-12320-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.60:46547
--webserver_interface=127.12.8.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.60.keytab
--principal=kudu/127.12.8.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:46233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.62:44539,127.12.8.61:36113,127.12.8.60:46547
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:18:32.110376 2316 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.8.60" port: 46547 } attempt: 1
I20260925 15:18:32.111624 2318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:32.111086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.60:46547 (local address 127.0.0.1:49024)
0925 15:18:32.111317 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:32.111426 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.60:46547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260925 15:18:32.112692 2318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:32.112278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.60:46547 (local address 127.0.0.1:49038)
0925 15:18:32.112430 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:32.112571 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.60:46547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260925 15:18:32.112965 2316 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.8.60:46547: Network error: Client connection negotiation failed: client connection to 127.12.8.60:46547: connect: Connection refused (error 111)
I20260925 15:18:32.181232 2316 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.8.60" port: 46547 } attempt: 2
I20260925 15:18:32.182492 2318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:32.181917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.60:46547 (local address 127.0.0.1:49054)
0925 15:18:32.182145 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:32.182305 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.60:46547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260925 15:18:32.183506 2318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:32.183185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.60:46547 (local address 127.0.0.1:49066)
0925 15:18:32.183323 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:32.183393 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.60:46547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260925 15:18:32.183775 2316 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.8.60:46547: Network error: Client connection negotiation failed: client connection to 127.12.8.60:46547: connect: Connection refused (error 111)
I20260925 15:18:32.203907 2242 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.8.61" port: 36113 } attempt: 5
Sep 25 15:18:32 dist-test-slave-p468 krb5kdc[2162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349511, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.62@KRBTEST.COM for kudu/127.12.8.61@KRBTEST.COM
I20260925 15:18:32.218710 2244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:32.204630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.61:36113 (local address 127.0.0.1:60476)
0925 15:18:32.204851 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:32.204874 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:32.204985 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:32.205383 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:32.205386 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:32.205751 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:18:32.205919 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:32.205926 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:32.207394 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:32.207398 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:32.208295 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:32.208308 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:32.209526 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:32.209531 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:32.209711 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:32.210415 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:18:32.210433 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:18:32.212423 (+ 1990us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:18:32.216450 (+ 4027us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:32.216454 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:32.216457 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:32.216842 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:32.217216 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:32.217219 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:32.217222 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:32.217345 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:32.217918 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:18:32.217921 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:18:32.218058 (+ 137us) client_negotiation.cc:769] Sending connection context
0925 15:18:32.218170 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 15:18:32.218446 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260925 15:18:32.218860 2320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:32.204724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60476 (local address 127.12.8.61:36113)
0925 15:18:32.204906 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:32.204912 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:32.204936 (+ 24us) server_negotiation.cc:407] Connection header received
0925 15:18:32.205115 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:32.205121 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:32.205225 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:32.205336 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:32.206086 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:32.207211 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:32.208459 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:32.209364 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:32.209511 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:32.212620 (+ 3109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:32.212627 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:32.212632 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:32.212670 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:32.216315 (+ 3645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:32.216959 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:32.216964 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:32.216967 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:32.217109 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:32.217446 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:32.217451 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:32.217454 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:32.217800 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:32.217944 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:32.218333 (+ 389us) server_negotiation.cc:299] Negotiation successful
0925 15:18:32.218550 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 15:18:32.219950 2242 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.8.60" port: 46547 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:18:32.221089 2244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:32.220641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.60:46547 (local address 127.0.0.1:49070)
0925 15:18:32.220829 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:32.220923 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.60:46547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260925 15:18:32.222255 2244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:32.221775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.60:46547 (local address 127.0.0.1:49074)
0925 15:18:32.221948 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:32.222066 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.60:46547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 15:18:32.222500 2242 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.8.60:46547: Network error: Client connection negotiation failed: client connection to 127.12.8.60:46547: connect: Connection refused (error 111)
I20260925 15:18:32.273736 2242 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.8.60" port: 46547 } attempt: 1
I20260925 15:18:32.275056 2244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:32.274401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.60:46547 (local address 127.0.0.1:49080)
0925 15:18:32.274653 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:32.274817 (+ 164us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.60:46547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260925 15:18:32.275030 2316 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.8.60" port: 46547 } attempt: 3
I20260925 15:18:32.276512 2244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:32.276142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.60:46547 (local address 127.0.0.1:49102)
0925 15:18:32.276335 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:32.276406 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.60:46547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260925 15:18:32.276608 2318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:32.276068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.60:46547 (local address 127.0.0.1:49086)
0925 15:18:32.276315 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:32.276429 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.60:46547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260925 15:18:32.276751 2242 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.8.60:46547: Network error: Client connection negotiation failed: client connection to 127.12.8.60:46547: connect: Connection refused (error 111)
I20260925 15:18:32.277678 2318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:32.277253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.60:46547 (local address 127.0.0.1:49110)
0925 15:18:32.277472 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:32.277549 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.60:46547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139}
W20260925 15:18:32.277905 2316 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.8.60:46547: Network error: Client connection negotiation failed: client connection to 127.12.8.60:46547: connect: Connection refused (error 111)
I20260925 15:18:32.344981 2242 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.8.60" port: 46547 } attempt: 2
I20260925 15:18:32.346288 2244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:32.345604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.60:46547 (local address 127.0.0.1:49114)
0925 15:18:32.345862 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:32.346010 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.60:46547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260925 15:18:32.347348 2244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:32.346965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.60:46547 (local address 127.0.0.1:49126)
0925 15:18:32.347115 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:32.347221 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.60:46547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260925 15:18:32.347568 2242 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.8.60:46547: Network error: Client connection negotiation failed: client connection to 127.12.8.60:46547: connect: Connection refused (error 111)
W20260925 15:18:32.395635 2323 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 15:18:32.395977 2323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:18:32.396098 2323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:18:32.396225 2323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:18:32.403275 2323 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:18:32.403362 2323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:18:32.403456 2323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:18:32.403533 2323 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:18:32.403620 2323 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:18:32.413326 2323 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:46233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349162370577-12320-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349162370577-12320-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.12.8.62:44539,127.12.8.61:36113,127.12.8.60:46547
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.60.keytab
--principal=kudu/127.12.8.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.60:46547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349162370577-12320-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.12.8.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349162370577-12320-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:18:32.414609 2323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:18:32.415906 2323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 15:18:32.421144 2316 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.8.60" port: 46547 } attempt: 4
I20260925 15:18:32.422467 2318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:32.421836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.60:46547 (local address 127.0.0.1:49132)
0925 15:18:32.422222 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:32.422329 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.60:46547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168}
I20260925 15:18:32.423409 2318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:32.423139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.60:46547 (local address 127.0.0.1:49136)
0925 15:18:32.423257 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:32.423320 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.60:46547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260925 15:18:32.423691 2316 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.8.60:46547: Network error: Client connection negotiation failed: client connection to 127.12.8.60:46547: connect: Connection refused (error 111)
W20260925 15:18:32.427915 2328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:32.427949 2329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:32.428133 2331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:18:32.428706 2323 server_base.cc:1034] running on GCE node
I20260925 15:18:32.429348 2323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:18:32.430667 2323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:18:32.431875 2323 hybrid_clock.cc:648] HybridClock initialized: now 1790349512431801 us; error 47 us; skew 500 ppm
I20260925 15:18:32.432186 2323 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:18:32 dist-test-slave-p468 krb5kdc[2162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349512, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:18:32.435390 2323 init.cc:377] Logged in from keytab as kudu/127.12.8.60@KRBTEST.COM (short username kudu)
I20260925 15:18:32.436849 2323 webserver.cc:460] Webserver started at http://127.12.8.60:35761/ using document root <none> and password file <none>
I20260925 15:18:32.437494 2323 fs_manager.cc:359] Metadata directory not provided
I20260925 15:18:32.437604 2323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:18:32.437934 2323 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:18:32.440837 2323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349162370577-12320-0/minicluster-data/master-2/data/instance:
uuid: "53448c30928348d29ecaee03698c7072"
format_stamp: "Formatted at 2026-09-25 15:18:32 on dist-test-slave-p468"
server_key: "ffa44f912af6ba7b0a2a2c17facb95f3"
server_key_iv: "3e50a2b5036d73b1146d6dcedafe4df4"
server_key_version: "encryptionkey@0"
I20260925 15:18:32.441586 2323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349162370577-12320-0/minicluster-data/master-2/wal/instance:
uuid: "53448c30928348d29ecaee03698c7072"
format_stamp: "Formatted at 2026-09-25 15:18:32 on dist-test-slave-p468"
server_key: "ffa44f912af6ba7b0a2a2c17facb95f3"
server_key_iv: "3e50a2b5036d73b1146d6dcedafe4df4"
server_key_version: "encryptionkey@0"
I20260925 15:18:32.446497 2323 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260925 15:18:32.449620 2338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:32.450938 2323 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:18:32.451114 2323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349162370577-12320-0/minicluster-data/master-2/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349162370577-12320-0/minicluster-data/master-2/wal
uuid: "53448c30928348d29ecaee03698c7072"
format_stamp: "Formatted at 2026-09-25 15:18:32 on dist-test-slave-p468"
server_key: "ffa44f912af6ba7b0a2a2c17facb95f3"
server_key_iv: "3e50a2b5036d73b1146d6dcedafe4df4"
server_key_version: "encryptionkey@0"
I20260925 15:18:32.451320 2323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349162370577-12320-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349162370577-12320-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349162370577-12320-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:18:32.453794 2242 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.8.60" port: 46547 } attempt: 3
I20260925 15:18:32.454928 2244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:32.454409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.60:46547 (local address 127.0.0.1:49150)
0925 15:18:32.454648 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:32.454751 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.60:46547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260925 15:18:32.456014 2244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:32.455602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.60:46547 (local address 127.0.0.1:49152)
0925 15:18:32.455797 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:32.455884 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.60:46547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
W20260925 15:18:32.456249 2242 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.8.60:46547: Network error: Client connection negotiation failed: client connection to 127.12.8.60:46547: connect: Connection refused (error 111)
I20260925 15:18:32.479112 2323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:18:32.482921 2323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:18:32.483139 2323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:18:32.501497 2323 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.60:46547
I20260925 15:18:32.501523 2389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.60:46547 every 8 connection(s)
I20260925 15:18:32.502924 2323 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349162370577-12320-0/minicluster-data/master-2/data/info.pb
I20260925 15:18:32.504231 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 2323
I20260925 15:18:32.504444 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349162370577-12320-0/minicluster-data/master-2/wal/instance
I20260925 15:18:32.505115 12320 external_mini_cluster.cc:1428] Setting key d58e65bb00dc90512000063dd0e1bfd9
I20260925 15:18:32.507507 2390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:18:32.517110 2390 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.8.62" port: 44539 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:18:32 dist-test-slave-p468 krb5kdc[2162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.60@KRBTEST.COM
Sep 25 15:18:32 dist-test-slave-p468 krb5kdc[2162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349512, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.60@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
I20260925 15:18:32.530344 2393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:32.507878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49166 (local address 127.12.8.60:46547)
0925 15:18:32.508976 (+ 1098us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:32.508991 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:32.509049 (+ 58us) server_negotiation.cc:407] Connection header received
0925 15:18:32.510972 (+ 1923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:32.510999 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:32.511514 (+ 515us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:32.511949 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:32.513225 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:32.515284 (+ 2059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:32.516550 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:32.517571 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:32.517753 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:32.522099 (+ 4346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:32.522106 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:32.522115 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:32.522151 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:32.525698 (+ 3547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:32.526476 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:32.526483 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:32.526489 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:32.526652 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:32.527087 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:32.527090 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:32.527093 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:32.527729 (+ 636us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:32.527965 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:32.528503 (+ 538us) server_negotiation.cc:299] Negotiation successful
0925 15:18:32.528934 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":855,"thread_start_us":600,"threads_started":1}
I20260925 15:18:32.534551 2394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:32.519849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:44539 (local address 127.0.0.1:60860)
0925 15:18:32.520468 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:32.520502 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:32.520672 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:32.521039 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:32.521046 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:32.521668 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:18:32.521890 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:32.521898 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:32.522971 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:32.522975 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:32.524179 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:32.524190 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:32.525200 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:32.525204 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:32.525322 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:32.525905 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:18:32.525926 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:18:32.528180 (+ 2254us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:18:32.531685 (+ 3505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:32.531700 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:32.531706 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:32.532255 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:32.532626 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:32.532631 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:32.532635 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:32.532812 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:32.533442 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:18:32.533451 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:18:32.533639 (+ 188us) client_negotiation.cc:769] Sending connection context
0925 15:18:32.533921 (+ 282us) client_negotiation.cc:240] Negotiation successful
0925 15:18:32.534170 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":304,"threads_started":1}
I20260925 15:18:32.534739 2245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:32.519877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60860 (local address 127.12.8.62:44539)
0925 15:18:32.520102 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:32.520110 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:32.520601 (+ 491us) server_negotiation.cc:407] Connection header received
0925 15:18:32.520798 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:32.520804 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:32.520873 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:32.520978 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:32.522052 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:32.522818 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:32.524325 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:32.525021 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:32.525171 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:32.528374 (+ 3203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:32.528381 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:32.528386 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:32.528425 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:32.531568 (+ 3143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:32.532378 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:32.532382 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:32.532384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:32.532510 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:32.532940 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:32.532944 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:32.532946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:32.533272 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:32.533458 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:32.534078 (+ 620us) server_negotiation.cc:299] Negotiation successful
0925 15:18:32.534321 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 15:18:32.536197 2390 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.8.61" port: 36113 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:18:32 dist-test-slave-p468 krb5kdc[2162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
Sep 25 15:18:32 dist-test-slave-p468 krb5kdc[2162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349512, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.60@KRBTEST.COM for kudu/127.12.8.61@KRBTEST.COM
I20260925 15:18:32.549631 2395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:32.533543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60868 (local address 127.12.8.62:44539)
0925 15:18:32.534165 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:32.534170 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:32.534218 (+ 48us) server_negotiation.cc:407] Connection header received
0925 15:18:32.534265 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:32.534271 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:32.534333 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:32.534438 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:32.535930 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:32.536929 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:32.538684 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:32.539303 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:32.539424 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:32.543764 (+ 4340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:32.543768 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:32.543772 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:32.543799 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:32.547091 (+ 3292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:32.547690 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:32.547696 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:32.547701 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:32.547843 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:32.548186 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:32.548192 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:32.548196 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:32.548452 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:32.548736 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:32.549008 (+ 272us) server_negotiation.cc:299] Negotiation successful
0925 15:18:32.549221 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":258,"threads_started":1}
I20260925 15:18:32.550494 2394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:32.536918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.61:36113 (local address 127.0.0.1:60478)
0925 15:18:32.537258 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:32.537277 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:32.537378 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:32.537670 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:32.537675 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:32.538106 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:18:32.538325 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:32.538337 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:32.539594 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:32.539622 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:32.541030 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:32.541047 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:32.541727 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:32.541734 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:32.541914 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:32.542517 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:18:32.542544 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:18:32.544634 (+ 2090us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:18:32.548124 (+ 3490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:32.548131 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:32.548135 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:32.548477 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:32.548856 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:32.548861 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:32.548864 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:32.548992 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:32.549520 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:18:32.549525 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:18:32.549704 (+ 179us) client_negotiation.cc:769] Sending connection context
0925 15:18:32.549893 (+ 189us) client_negotiation.cc:240] Negotiation successful
0925 15:18:32.550129 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260925 15:18:32.550521 2320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:32.536952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60478 (local address 127.12.8.61:36113)
0925 15:18:32.537137 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:32.537145 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:32.537351 (+ 206us) server_negotiation.cc:407] Connection header received
0925 15:18:32.537478 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:32.537482 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:32.537541 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:32.537618 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:32.538457 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:32.539435 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:32.541186 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:32.541577 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:32.541677 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:32.544809 (+ 3132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:32.544815 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:32.544820 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:32.544862 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:32.548009 (+ 3147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:32.548592 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:32.548599 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:32.548602 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:32.548746 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:32.549095 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:32.549102 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:32.549105 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:32.549376 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:32.549540 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:32.550018 (+ 478us) server_negotiation.cc:299] Negotiation successful
0925 15:18:32.550274 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 15:18:32.551594 2390 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.8.60" port: 46547 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:18:32 dist-test-slave-p468 krb5kdc[2162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349512, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.60@KRBTEST.COM for kudu/127.12.8.60@KRBTEST.COM
Sep 25 15:18:32 dist-test-slave-p468 krb5kdc[2162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.61@KRBTEST.COM
I20260925 15:18:32.565543 2394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:32.552418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.60:46547 (local address 127.0.0.1:49174)
0925 15:18:32.552550 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:32.552565 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:32.552688 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:32.552968 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:32.552972 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:32.553405 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:18:32.553561 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:32.553582 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:32.554691 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:32.554696 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:32.555705 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:32.555737 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:32.556448 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:32.556454 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:32.556725 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:32.557159 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:18:32.557186 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:18:32.559785 (+ 2599us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:18:32.563027 (+ 3242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:32.563031 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:32.563035 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:32.563577 (+ 542us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:32.563937 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:32.563943 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:32.563946 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:32.564084 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:32.564550 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:18:32.564554 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:18:32.564699 (+ 145us) client_negotiation.cc:769] Sending connection context
0925 15:18:32.564830 (+ 131us) client_negotiation.cc:240] Negotiation successful
0925 15:18:32.565131 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260925 15:18:32.565649 2393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:32.552335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49174 (local address 127.12.8.60:46547)
0925 15:18:32.552509 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:32.552515 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:32.552640 (+ 125us) server_negotiation.cc:407] Connection header received
0925 15:18:32.552803 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:32.552807 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:32.552866 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:32.552955 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:32.553715 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:32.554551 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:32.555836 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:32.556303 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:32.556433 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:32.559975 (+ 3542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:32.559980 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:32.559983 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:32.560009 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:32.562894 (+ 2885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:32.563702 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:32.563705 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:32.563707 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:32.563825 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:32.564190 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:32.564197 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:32.564205 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:32.564441 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:32.564595 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:32.564918 (+ 323us) server_negotiation.cc:299] Negotiation successful
0925 15:18:32.565289 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":150,"server-negotiator.queue_time_us":46}
I20260925 15:18:32.566706 2320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:32.551384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60488 (local address 127.12.8.61:36113)
0925 15:18:32.551530 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:32.551535 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:32.551644 (+ 109us) server_negotiation.cc:407] Connection header received
0925 15:18:32.551822 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:32.551827 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:32.551894 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:32.551994 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:32.553273 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:32.554118 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:32.555468 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:32.555898 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:32.556022 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:32.560770 (+ 4748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:32.560774 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:32.560777 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:32.560805 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:32.563793 (+ 2988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:32.564415 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:32.564420 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:32.564424 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:32.564557 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:32.564927 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:32.564934 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:32.564937 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:32.565191 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:32.565446 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:32.565726 (+ 280us) server_negotiation.cc:299] Negotiation successful
0925 15:18:32.565928 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 15:18:32.571483 12320 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:18:32.571718 2390 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53448c30928348d29ecaee03698c7072: Bootstrap starting.
I20260925 15:18:32.575472 2390 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 53448c30928348d29ecaee03698c7072: Neither blocks nor log segments found. Creating new log.
I20260925 15:18:32.576489 2390 log.cc:824] T 00000000000000000000000000000000 P 53448c30928348d29ecaee03698c7072: Log is configured to *not* fsync() on all Append() calls
I20260925 15:18:32.574869 2217 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "a93fde3803b04be9b5f51407e2102ac5"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60868
I20260925 15:18:32.579212 2390 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53448c30928348d29ecaee03698c7072: No bootstrap required, opened a new log
I20260925 15:18:32.585918 2390 raft_consensus.cc:348] T 00000000000000000000000000000000 P 53448c30928348d29ecaee03698c7072 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a93fde3803b04be9b5f51407e2102ac5" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 44539 } } peers { permanent_uuid: "a7db4ff52dca4ab08b2a471c0c94df52" member_type: VOTER last_known_addr { host: "127.12.8.61" port: 36113 } } peers { permanent_uuid: "53448c30928348d29ecaee03698c7072" member_type: VOTER last_known_addr { host: "127.12.8.60" port: 46547 } }
I20260925 15:18:32.586242 2390 raft_consensus.cc:729] T 00000000000000000000000000000000 P 53448c30928348d29ecaee03698c7072 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53448c30928348d29ecaee03698c7072, State: Initialized, Role: FOLLOWER
I20260925 15:18:32.586701 2390 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53448c30928348d29ecaee03698c7072 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a93fde3803b04be9b5f51407e2102ac5" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 44539 } } peers { permanent_uuid: "a7db4ff52dca4ab08b2a471c0c94df52" member_type: VOTER last_known_addr { host: "127.12.8.61" port: 36113 } } peers { permanent_uuid: "53448c30928348d29ecaee03698c7072" member_type: VOTER last_known_addr { host: "127.12.8.60" port: 46547 } }
I20260925 15:18:32.587493 2397 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53448c30928348d29ecaee03698c7072 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a93fde3803b04be9b5f51407e2102ac5" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 44539 } } peers { permanent_uuid: "a7db4ff52dca4ab08b2a471c0c94df52" member_type: VOTER last_known_addr { host: "127.12.8.61" port: 36113 } } peers { permanent_uuid: "53448c30928348d29ecaee03698c7072" member_type: VOTER last_known_addr { host: "127.12.8.60" port: 46547 } } }
I20260925 15:18:32.587831 2397 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53448c30928348d29ecaee03698c7072 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:18:32.588738 2390 sys_catalog.cc:564] T 00000000000000000000000000000000 P 53448c30928348d29ecaee03698c7072 [sys.catalog]: configured and running, proceeding with master startup.
W20260925 15:18:32.597759 2408 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 53448c30928348d29ecaee03698c7072: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:18:32.597878 2408 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:18:32.595721 2323 master_runner.cc:186] Error getting master registration for 127.12.8.62:44539: OK, instance_id { permanent_uuid: "a93fde3803b04be9b5f51407e2102ac5" instance_seqno: 1790349511493686 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 15:18:32.603636 2323 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 15:18:32.633493 2242 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.8.60" port: 46547 } attempt: 4
Sep 25 15:18:32 dist-test-slave-p468 krb5kdc[2162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349511, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.62@KRBTEST.COM for kudu/127.12.8.60@KRBTEST.COM
I20260925 15:18:32.649392 2244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:32.634369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.60:46547 (local address 127.0.0.1:49178)
0925 15:18:32.634602 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:32.634625 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:32.634762 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:32.635121 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:32.635127 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:32.635629 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:18:32.635850 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:32.635863 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:32.637301 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:32.637310 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:32.638845 (+ 1535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:32.638860 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:32.639859 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:32.639865 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:32.640062 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:32.640757 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:18:32.640781 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:18:32.643705 (+ 2924us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:18:32.646981 (+ 3276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:32.646986 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:32.646990 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:32.647346 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:32.647723 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:32.647729 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:32.647733 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:32.647882 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:32.648411 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:18:32.648416 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:18:32.648574 (+ 158us) client_negotiation.cc:769] Sending connection context
0925 15:18:32.648718 (+ 144us) client_negotiation.cc:240] Negotiation successful
0925 15:18:32.649015 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20260925 15:18:32.649397 2393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:32.634599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49178 (local address 127.12.8.60:46547)
0925 15:18:32.634811 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:32.634815 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:32.634852 (+ 37us) server_negotiation.cc:407] Connection header received
0925 15:18:32.634908 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:32.634912 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:32.634970 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:32.635083 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:32.636002 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:32.637176 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:32.638989 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:32.639705 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:32.639826 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:32.643900 (+ 4074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:32.643905 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:32.643908 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:32.643936 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:32.646811 (+ 2875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:32.647469 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:32.647474 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:32.647488 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:32.647604 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:32.648003 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:32.648007 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:32.648009 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:32.648295 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:32.648466 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:32.648825 (+ 359us) server_negotiation.cc:299] Negotiation successful
0925 15:18:32.649020 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 15:18:32.655539 2242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a93fde3803b04be9b5f51407e2102ac5: Bootstrap starting.
I20260925 15:18:32.659694 2242 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a93fde3803b04be9b5f51407e2102ac5: Neither blocks nor log segments found. Creating new log.
I20260925 15:18:32.661020 2242 log.cc:824] T 00000000000000000000000000000000 P a93fde3803b04be9b5f51407e2102ac5: Log is configured to *not* fsync() on all Append() calls
I20260925 15:18:32.664513 2242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a93fde3803b04be9b5f51407e2102ac5: No bootstrap required, opened a new log
I20260925 15:18:32.672178 2242 raft_consensus.cc:348] T 00000000000000000000000000000000 P a93fde3803b04be9b5f51407e2102ac5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a93fde3803b04be9b5f51407e2102ac5" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 44539 } } peers { permanent_uuid: "a7db4ff52dca4ab08b2a471c0c94df52" member_type: VOTER last_known_addr { host: "127.12.8.61" port: 36113 } } peers { permanent_uuid: "53448c30928348d29ecaee03698c7072" member_type: VOTER last_known_addr { host: "127.12.8.60" port: 46547 } }
I20260925 15:18:32.672601 2242 raft_consensus.cc:729] T 00000000000000000000000000000000 P a93fde3803b04be9b5f51407e2102ac5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a93fde3803b04be9b5f51407e2102ac5, State: Initialized, Role: FOLLOWER
I20260925 15:18:32.673122 2242 consensus_queue.cc:260] T 00000000000000000000000000000000 P a93fde3803b04be9b5f51407e2102ac5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a93fde3803b04be9b5f51407e2102ac5" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 44539 } } peers { permanent_uuid: "a7db4ff52dca4ab08b2a471c0c94df52" member_type: VOTER last_known_addr { host: "127.12.8.61" port: 36113 } } peers { permanent_uuid: "53448c30928348d29ecaee03698c7072" member_type: VOTER last_known_addr { host: "127.12.8.60" port: 46547 } }
I20260925 15:18:32.674118 2411 sys_catalog.cc:455] T 00000000000000000000000000000000 P a93fde3803b04be9b5f51407e2102ac5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a93fde3803b04be9b5f51407e2102ac5" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 44539 } } peers { permanent_uuid: "a7db4ff52dca4ab08b2a471c0c94df52" member_type: VOTER last_known_addr { host: "127.12.8.61" port: 36113 } } peers { permanent_uuid: "53448c30928348d29ecaee03698c7072" member_type: VOTER last_known_addr { host: "127.12.8.60" port: 46547 } } }
I20260925 15:18:32.674541 2411 sys_catalog.cc:458] T 00000000000000000000000000000000 P a93fde3803b04be9b5f51407e2102ac5 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:18:32.675488 2242 sys_catalog.cc:564] T 00000000000000000000000000000000 P a93fde3803b04be9b5f51407e2102ac5 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:18:32.675537 2217 raft_consensus.cc:482] T 00000000000000000000000000000000 P a93fde3803b04be9b5f51407e2102ac5 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 15:18:32.675740 2217 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a93fde3803b04be9b5f51407e2102ac5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:18:32.678356 2217 raft_consensus.cc:504] T 00000000000000000000000000000000 P a93fde3803b04be9b5f51407e2102ac5 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a93fde3803b04be9b5f51407e2102ac5" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 44539 } } peers { permanent_uuid: "a7db4ff52dca4ab08b2a471c0c94df52" member_type: VOTER last_known_addr { host: "127.12.8.61" port: 36113 } } peers { permanent_uuid: "53448c30928348d29ecaee03698c7072" member_type: VOTER last_known_addr { host: "127.12.8.60" port: 46547 } }
I20260925 15:18:32.680972 2365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a93fde3803b04be9b5f51407e2102ac5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "53448c30928348d29ecaee03698c7072"
I20260925 15:18:32.681505 2365 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 53448c30928348d29ecaee03698c7072 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:18:32.685022 2365 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 53448c30928348d29ecaee03698c7072 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a93fde3803b04be9b5f51407e2102ac5 in term 1.
I20260925 15:18:32.685688 2194 leader_election.cc:302] T 00000000000000000000000000000000 P a93fde3803b04be9b5f51407e2102ac5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53448c30928348d29ecaee03698c7072, a93fde3803b04be9b5f51407e2102ac5; no voters:
I20260925 15:18:32.686178 2411 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a93fde3803b04be9b5f51407e2102ac5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:18:32.687028 2411 raft_consensus.cc:686] T 00000000000000000000000000000000 P a93fde3803b04be9b5f51407e2102ac5 [term 1 LEADER]: Becoming Leader. State: Replica: a93fde3803b04be9b5f51407e2102ac5, State: Running, Role: LEADER
I20260925 15:18:32.687606 2411 consensus_queue.cc:237] T 00000000000000000000000000000000 P a93fde3803b04be9b5f51407e2102ac5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a93fde3803b04be9b5f51407e2102ac5" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 44539 } } peers { permanent_uuid: "a7db4ff52dca4ab08b2a471c0c94df52" member_type: VOTER last_known_addr { host: "127.12.8.61" port: 36113 } } peers { permanent_uuid: "53448c30928348d29ecaee03698c7072" member_type: VOTER last_known_addr { host: "127.12.8.60" port: 46547 } }
I20260925 15:18:32.689994 2217 leader_election.cc:288] T 00000000000000000000000000000000 P a93fde3803b04be9b5f51407e2102ac5 [CANDIDATE]: Term 1 election: Requested vote from peers a7db4ff52dca4ab08b2a471c0c94df52 (127.12.8.61:36113), 53448c30928348d29ecaee03698c7072 (127.12.8.60:46547)
I20260925 15:18:32.693281 2418 sys_catalog.cc:455] T 00000000000000000000000000000000 P a93fde3803b04be9b5f51407e2102ac5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a93fde3803b04be9b5f51407e2102ac5. Latest consensus state: current_term: 1 leader_uuid: "a93fde3803b04be9b5f51407e2102ac5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a93fde3803b04be9b5f51407e2102ac5" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 44539 } } peers { permanent_uuid: "a7db4ff52dca4ab08b2a471c0c94df52" member_type: VOTER last_known_addr { host: "127.12.8.61" port: 36113 } } peers { permanent_uuid: "53448c30928348d29ecaee03698c7072" member_type: VOTER last_known_addr { host: "127.12.8.60" port: 46547 } } }
I20260925 15:18:32.693760 2418 sys_catalog.cc:458] T 00000000000000000000000000000000 P a93fde3803b04be9b5f51407e2102ac5 [sys.catalog]: This master's current role is: LEADER
W20260925 15:18:32.721508 2427 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a93fde3803b04be9b5f51407e2102ac5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:18:32.721818 2427 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:18:32.722028 2423 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:18:32.711629 2175 master_runner.cc:186] Error getting master registration for 127.12.8.61:36113: OK, instance_id { permanent_uuid: "a7db4ff52dca4ab08b2a471c0c94df52" instance_seqno: 1790349511992372 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 15:18:32.723495 2175 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 15:18:32.724269 2423 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:18:32.728186 2316 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.8.60" port: 46547 } attempt: 5
W20260925 15:18:32.733108 2196 tablet.cc:2369] T 00000000000000000000000000000000 P a93fde3803b04be9b5f51407e2102ac5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:18:32.735975 2344 tablet.cc:2369] T 00000000000000000000000000000000 P 53448c30928348d29ecaee03698c7072: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 15:18:32 dist-test-slave-p468 krb5kdc[2162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349511, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.61@KRBTEST.COM for kudu/127.12.8.60@KRBTEST.COM
I20260925 15:18:32.747853 2318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:32.728846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.60:46547 (local address 127.0.0.1:49192)
0925 15:18:32.729114 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:32.729135 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:32.729268 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:32.729638 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:32.729643 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:32.730224 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:18:32.730457 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:32.730470 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:32.731996 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:32.732002 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:32.733455 (+ 1453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:32.733471 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:32.734846 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:32.734854 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:32.735045 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:32.735828 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:18:32.735851 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:18:32.739007 (+ 3156us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:18:32.743805 (+ 4798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:32.743810 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:32.743815 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:32.744341 (+ 526us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:32.744977 (+ 636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:32.744983 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:32.744987 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:32.745243 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:32.746395 (+ 1152us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:18:32.746401 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:18:32.746641 (+ 240us) client_negotiation.cc:769] Sending connection context
0925 15:18:32.746845 (+ 204us) client_negotiation.cc:240] Negotiation successful
0925 15:18:32.747279 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149}
I20260925 15:18:32.749763 2393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:32.728920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49192 (local address 127.12.8.60:46547)
0925 15:18:32.729112 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:32.729119 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:32.729211 (+ 92us) server_negotiation.cc:407] Connection header received
0925 15:18:32.729397 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:32.729403 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:32.729493 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:32.729617 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:32.730607 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:32.731789 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:32.733681 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:32.734495 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:32.734617 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:32.739328 (+ 4711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:32.739335 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:32.739340 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:32.739379 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:32.743641 (+ 4262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:32.744528 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:32.744537 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:32.744540 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:32.744802 (+ 262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:32.745480 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:32.745487 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:32.745490 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:32.746173 (+ 683us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:32.748490 (+ 2317us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:32.748685 (+ 195us) server_negotiation.cc:299] Negotiation successful
0925 15:18:32.749062 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 15:18:32.756047 2316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7db4ff52dca4ab08b2a471c0c94df52: Bootstrap starting.
I20260925 15:18:32.760977 2316 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a7db4ff52dca4ab08b2a471c0c94df52: Neither blocks nor log segments found. Creating new log.
I20260925 15:18:32.763413 2316 log.cc:824] T 00000000000000000000000000000000 P a7db4ff52dca4ab08b2a471c0c94df52: Log is configured to *not* fsync() on all Append() calls
I20260925 15:18:32.774631 2395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:32.732452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60882 (local address 127.12.8.62:44539)
0925 15:18:32.732621 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:32.732629 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:32.742775 (+ 10146us) server_negotiation.cc:407] Connection header received
0925 15:18:32.743192 (+ 417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:32.743198 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:32.743313 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:32.743384 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:32.754413 (+ 11029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:32.755562 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:32.758875 (+ 3313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:32.759638 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:32.759738 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:32.765566 (+ 5828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:32.765573 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:32.765578 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:32.765622 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:32.769731 (+ 4109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:32.771163 (+ 1432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:32.771169 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:32.771173 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:32.771354 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:32.772020 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:32.772026 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:32.772030 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:32.772404 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:32.772635 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:32.773585 (+ 950us) server_negotiation.cc:299] Negotiation successful
0925 15:18:32.773976 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 15:18:32.775848 2365 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 53448c30928348d29ecaee03698c7072 [term 1 FOLLOWER]: Refusing update from remote peer a93fde3803b04be9b5f51407e2102ac5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:18:32.777653 2418 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a93fde3803b04be9b5f51407e2102ac5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53448c30928348d29ecaee03698c7072" member_type: VOTER last_known_addr { host: "127.12.8.60" port: 46547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:18:32.785164 2316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7db4ff52dca4ab08b2a471c0c94df52: No bootstrap required, opened a new log
I20260925 15:18:32.797297 2320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:32.732111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60492 (local address 127.12.8.61:36113)
0925 15:18:32.732292 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:32.732297 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:32.761887 (+ 29590us) server_negotiation.cc:407] Connection header received
0925 15:18:32.764341 (+ 2454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:32.764348 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:32.764430 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:32.764496 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:32.770732 (+ 6236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:32.771847 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:32.778336 (+ 6489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:32.778998 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:32.779105 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:32.786318 (+ 7213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:32.786327 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:32.786334 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:32.786389 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:32.791298 (+ 4909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:32.792922 (+ 1624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:32.792940 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:32.792943 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:32.793106 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:32.793895 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:32.793900 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:32.793904 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:32.794293 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:32.794535 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:32.796098 (+ 1563us) server_negotiation.cc:299] Negotiation successful
0925 15:18:32.796535 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 15:18:32.803560 2393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:32.734564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49202 (local address 127.12.8.60:46547)
0925 15:18:32.751549 (+ 16985us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:32.751646 (+ 97us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:32.764206 (+ 12560us) server_negotiation.cc:407] Connection header received
0925 15:18:32.764423 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:32.764428 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:32.764504 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:32.764570 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:32.775321 (+ 10751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:32.776430 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:32.782362 (+ 5932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:32.783082 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:32.783180 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:32.786593 (+ 3413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:32.786599 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:32.786603 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:32.786644 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:32.790754 (+ 4110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:32.798647 (+ 7893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:32.798652 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:32.798664 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:32.798813 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:32.801127 (+ 2314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:32.801133 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:32.801137 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:32.801485 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:32.801733 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:32.802856 (+ 1123us) server_negotiation.cc:299] Negotiation successful
0925 15:18:32.803105 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16563,"thread_start_us":27612,"threads_started":1}
I20260925 15:18:32.807515 2397 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53448c30928348d29ecaee03698c7072 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a93fde3803b04be9b5f51407e2102ac5. Latest consensus state: current_term: 1 leader_uuid: "a93fde3803b04be9b5f51407e2102ac5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a93fde3803b04be9b5f51407e2102ac5" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 44539 } } peers { permanent_uuid: "a7db4ff52dca4ab08b2a471c0c94df52" member_type: VOTER last_known_addr { host: "127.12.8.61" port: 36113 } } peers { permanent_uuid: "53448c30928348d29ecaee03698c7072" member_type: VOTER last_known_addr { host: "127.12.8.60" port: 46547 } } }
I20260925 15:18:32.809401 2397 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53448c30928348d29ecaee03698c7072 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:18:32.811496 2431 mvcc.cc:205] Tried to move back new op lower bound from 7333271604290547712 to 7333271603973402624. Current Snapshot: MvccSnapshot[applied={T|T < 7333271604290547712}]
I20260925 15:18:32.811815 2316 raft_consensus.cc:348] T 00000000000000000000000000000000 P a7db4ff52dca4ab08b2a471c0c94df52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a93fde3803b04be9b5f51407e2102ac5" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 44539 } } peers { permanent_uuid: "a7db4ff52dca4ab08b2a471c0c94df52" member_type: VOTER last_known_addr { host: "127.12.8.61" port: 36113 } } peers { permanent_uuid: "53448c30928348d29ecaee03698c7072" member_type: VOTER last_known_addr { host: "127.12.8.60" port: 46547 } }
I20260925 15:18:32.812994 2316 raft_consensus.cc:729] T 00000000000000000000000000000000 P a7db4ff52dca4ab08b2a471c0c94df52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7db4ff52dca4ab08b2a471c0c94df52, State: Initialized, Role: FOLLOWER
I20260925 15:18:32.813683 2316 consensus_queue.cc:260] T 00000000000000000000000000000000 P a7db4ff52dca4ab08b2a471c0c94df52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a93fde3803b04be9b5f51407e2102ac5" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 44539 } } peers { permanent_uuid: "a7db4ff52dca4ab08b2a471c0c94df52" member_type: VOTER last_known_addr { host: "127.12.8.61" port: 36113 } } peers { permanent_uuid: "53448c30928348d29ecaee03698c7072" member_type: VOTER last_known_addr { host: "127.12.8.60" port: 46547 } }
I20260925 15:18:32.815294 2418 sys_catalog.cc:455] T 00000000000000000000000000000000 P a93fde3803b04be9b5f51407e2102ac5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a93fde3803b04be9b5f51407e2102ac5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a93fde3803b04be9b5f51407e2102ac5" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 44539 } } peers { permanent_uuid: "a7db4ff52dca4ab08b2a471c0c94df52" member_type: VOTER last_known_addr { host: "127.12.8.61" port: 36113 } } peers { permanent_uuid: "53448c30928348d29ecaee03698c7072" member_type: VOTER last_known_addr { host: "127.12.8.60" port: 46547 } } }
I20260925 15:18:32.815565 2418 sys_catalog.cc:458] T 00000000000000000000000000000000 P a93fde3803b04be9b5f51407e2102ac5 [sys.catalog]: This master's current role is: LEADER
I20260925 15:18:32.823071 2423 catalog_manager.cc:1347] Generated new cluster ID: 01af0d47aadd4836ba03cdd24a3a580f
I20260925 15:18:32.823215 2423 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:18:32.835973 2397 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53448c30928348d29ecaee03698c7072 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a93fde3803b04be9b5f51407e2102ac5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a93fde3803b04be9b5f51407e2102ac5" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 44539 } } peers { permanent_uuid: "a7db4ff52dca4ab08b2a471c0c94df52" member_type: VOTER last_known_addr { host: "127.12.8.61" port: 36113 } } peers { permanent_uuid: "53448c30928348d29ecaee03698c7072" member_type: VOTER last_known_addr { host: "127.12.8.60" port: 46547 } } }
I20260925 15:18:32.836241 2397 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53448c30928348d29ecaee03698c7072 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:18:32.850777 2440 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7db4ff52dca4ab08b2a471c0c94df52 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a93fde3803b04be9b5f51407e2102ac5" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 44539 } } peers { permanent_uuid: "a7db4ff52dca4ab08b2a471c0c94df52" member_type: VOTER last_known_addr { host: "127.12.8.61" port: 36113 } } peers { permanent_uuid: "53448c30928348d29ecaee03698c7072" member_type: VOTER last_known_addr { host: "127.12.8.60" port: 46547 } } }
I20260925 15:18:32.851276 2440 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7db4ff52dca4ab08b2a471c0c94df52 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:18:32.852501 2316 sys_catalog.cc:564] T 00000000000000000000000000000000 P a7db4ff52dca4ab08b2a471c0c94df52 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:18:32.852600 2291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a93fde3803b04be9b5f51407e2102ac5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a7db4ff52dca4ab08b2a471c0c94df52"
I20260925 15:18:32.852937 2290 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a7db4ff52dca4ab08b2a471c0c94df52 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:18:32.856748 2290 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a7db4ff52dca4ab08b2a471c0c94df52 [term 1 FOLLOWER]: Refusing update from remote peer a93fde3803b04be9b5f51407e2102ac5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:18:32.860030 2291 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a7db4ff52dca4ab08b2a471c0c94df52 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a93fde3803b04be9b5f51407e2102ac5 in term 1.
I20260925 15:18:32.865329 2411 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a93fde3803b04be9b5f51407e2102ac5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7db4ff52dca4ab08b2a471c0c94df52" member_type: VOTER last_known_addr { host: "127.12.8.61" port: 36113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:18:32.885983 2423 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:18:32.888124 2423 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:18:32.911639 2423 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a93fde3803b04be9b5f51407e2102ac5: Generated new TSK 0
I20260925 15:18:32.912845 2423 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:18:32.947101 2440 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7db4ff52dca4ab08b2a471c0c94df52 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a93fde3803b04be9b5f51407e2102ac5. Latest consensus state: current_term: 1 leader_uuid: "a93fde3803b04be9b5f51407e2102ac5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a93fde3803b04be9b5f51407e2102ac5" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 44539 } } peers { permanent_uuid: "a7db4ff52dca4ab08b2a471c0c94df52" member_type: VOTER last_known_addr { host: "127.12.8.61" port: 36113 } } peers { permanent_uuid: "53448c30928348d29ecaee03698c7072" member_type: VOTER last_known_addr { host: "127.12.8.60" port: 46547 } } }
I20260925 15:18:32.947484 2440 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7db4ff52dca4ab08b2a471c0c94df52 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:18:32.948107 2454 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7db4ff52dca4ab08b2a471c0c94df52 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a93fde3803b04be9b5f51407e2102ac5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a93fde3803b04be9b5f51407e2102ac5" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 44539 } } peers { permanent_uuid: "a7db4ff52dca4ab08b2a471c0c94df52" member_type: VOTER last_known_addr { host: "127.12.8.61" port: 36113 } } peers { permanent_uuid: "53448c30928348d29ecaee03698c7072" member_type: VOTER last_known_addr { host: "127.12.8.60" port: 46547 } } }
I20260925 15:18:32.948534 2454 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7db4ff52dca4ab08b2a471c0c94df52 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:18:32.956966 2411 sys_catalog.cc:455] T 00000000000000000000000000000000 P a93fde3803b04be9b5f51407e2102ac5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a93fde3803b04be9b5f51407e2102ac5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a93fde3803b04be9b5f51407e2102ac5" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 44539 } } peers { permanent_uuid: "a7db4ff52dca4ab08b2a471c0c94df52" member_type: VOTER last_known_addr { host: "127.12.8.61" port: 36113 } } peers { permanent_uuid: "53448c30928348d29ecaee03698c7072" member_type: VOTER last_known_addr { host: "127.12.8.60" port: 46547 } } }
I20260925 15:18:32.957425 2411 sys_catalog.cc:458] T 00000000000000000000000000000000 P a93fde3803b04be9b5f51407e2102ac5 [sys.catalog]: This master's current role is: LEADER
I20260925 15:18:32.958686 2457 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260925 15:18:33.045253 2395 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:18:33.031942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60896 (local address 127.12.8.62:44539)
0925 15:18:33.032155 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:33.032163 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:33.042860 (+ 10697us) server_negotiation.cc:407] Connection header received
0925 15:18:33.043435 (+ 575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:33.043442 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:33.043530 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:33.043618 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:33.044931 (+ 1313us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60896: BlockingRecv error: recv got EOF from 127.0.0.1:60896 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260925 15:18:33.047019 2320 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:18:33.031555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60504 (local address 127.12.8.61:36113)
0925 15:18:33.031761 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:33.031768 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:33.044846 (+ 13078us) server_negotiation.cc:407] Connection header received
0925 15:18:33.045311 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:33.045318 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:33.045420 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:33.045497 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:33.046784 (+ 1287us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60504: BlockingRecv error: recv got EOF from 127.0.0.1:60504 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260925 15:18:33.048687 2393 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:18:33.032162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49208 (local address 127.12.8.60:46547)
0925 15:18:33.032344 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:33.032350 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:33.046311 (+ 13961us) server_negotiation.cc:407] Connection header received
0925 15:18:33.046812 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:33.046818 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:33.046924 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:33.046998 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:33.048405 (+ 1407us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49208: BlockingRecv error: recv got EOF from 127.0.0.1:49208 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260925 15:18:33.055564 2320 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:18:33.053355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60516 (local address 127.12.8.61:36113)
0925 15:18:33.053530 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:33.053535 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:33.054050 (+ 515us) server_negotiation.cc:407] Connection header received
0925 15:18:33.054299 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:33.054308 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:33.054430 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:33.054560 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:33.055301 (+ 741us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60516: BlockingRecv error: recv got EOF from 127.0.0.1:60516 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260925 15:18:33.064857 2320 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:18:33.062680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60526 (local address 127.12.8.61:36113)
0925 15:18:33.062850 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:33.062856 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:33.063353 (+ 497us) server_negotiation.cc:407] Connection header received
0925 15:18:33.063597 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:33.063604 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:33.063696 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:33.063810 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:33.064643 (+ 833us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60526: BlockingRecv error: recv got EOF from 127.0.0.1:60526 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260925 15:18:33.073444 2320 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:18:33.071136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60542 (local address 127.12.8.61:36113)
0925 15:18:33.071319 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:33.071325 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:33.071791 (+ 466us) server_negotiation.cc:407] Connection header received
0925 15:18:33.072067 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:33.072074 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:33.072178 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:33.072254 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:33.073208 (+ 954us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60542: BlockingRecv error: recv got EOF from 127.0.0.1:60542 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260925 15:18:33.083854 2320 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:18:33.081628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60552 (local address 127.12.8.61:36113)
0925 15:18:33.081843 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:33.081850 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:33.082407 (+ 557us) server_negotiation.cc:407] Connection header received
0925 15:18:33.082604 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:33.082611 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:33.082703 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:33.082822 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:33.083632 (+ 810us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60552: BlockingRecv error: recv got EOF from 127.0.0.1:60552 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
W20260925 15:18:33.102878 2320 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:18:33.100421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60562 (local address 127.12.8.61:36113)
0925 15:18:33.100638 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:33.100646 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:33.101194 (+ 548us) server_negotiation.cc:407] Connection header received
0925 15:18:33.101424 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:33.101431 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:33.101551 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:33.101690 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:33.102627 (+ 937us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60562: BlockingRecv error: recv got EOF from 127.0.0.1:60562 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260925 15:18:33.130710 2320 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:18:33.128151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60572 (local address 127.12.8.61:36113)
0925 15:18:33.128353 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:33.128360 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:33.128961 (+ 601us) server_negotiation.cc:407] Connection header received
0925 15:18:33.129187 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:33.129197 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:33.129308 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:33.129439 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:33.130422 (+ 983us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60572: BlockingRecv error: recv got EOF from 127.0.0.1:60572 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260925 15:18:33.170351 2320 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:18:33.168025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60584 (local address 127.12.8.61:36113)
0925 15:18:33.168224 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:33.168231 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:33.168750 (+ 519us) server_negotiation.cc:407] Connection header received
0925 15:18:33.168956 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:33.168963 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:33.169069 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:33.169191 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:33.170033 (+ 842us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60584: BlockingRecv error: recv got EOF from 127.0.0.1:60584 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260925 15:18:33.240257 2320 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:18:33.238240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60592 (local address 127.12.8.61:36113)
0925 15:18:33.238450 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:33.238456 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:33.238838 (+ 382us) server_negotiation.cc:407] Connection header received
0925 15:18:33.238998 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:33.239005 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:33.239112 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:33.239245 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:33.239990 (+ 745us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60592: BlockingRecv error: recv got EOF from 127.0.0.1:60592 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260925 15:18:33.377856 2320 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:18:33.375864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37450 (local address 127.12.8.61:36113)
0925 15:18:33.376134 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:33.376143 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:33.376458 (+ 315us) server_negotiation.cc:407] Connection header received
0925 15:18:33.376702 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:33.376710 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:33.376792 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:33.376903 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:33.377650 (+ 747us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37450: BlockingRecv error: recv got EOF from 127.0.0.1:37450 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 15:18:33.600235 2408 catalog_manager.cc:1259] Loaded cluster ID: 01af0d47aadd4836ba03cdd24a3a580f
I20260925 15:18:33.600332 2408 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 53448c30928348d29ecaee03698c7072: loading cluster ID for follower catalog manager: success
I20260925 15:18:33.602799 2408 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 53448c30928348d29ecaee03698c7072: acquiring CA information for follower catalog manager: success
I20260925 15:18:33.604502 2408 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 53448c30928348d29ecaee03698c7072: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260925 15:18:33.642344 2320 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:18:33.640123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37466 (local address 127.12.8.61:36113)
0925 15:18:33.640393 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:33.640401 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:33.640674 (+ 273us) server_negotiation.cc:407] Connection header received
0925 15:18:33.640965 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:33.640970 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:33.641053 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:33.641186 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:33.642094 (+ 908us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37466: BlockingRecv error: recv got EOF from 127.0.0.1:37466 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
I20260925 15:18:33.963035 2457 catalog_manager.cc:1259] Loaded cluster ID: 01af0d47aadd4836ba03cdd24a3a580f
I20260925 15:18:33.963212 2457 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a7db4ff52dca4ab08b2a471c0c94df52: loading cluster ID for follower catalog manager: success
I20260925 15:18:33.965917 2457 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a7db4ff52dca4ab08b2a471c0c94df52: acquiring CA information for follower catalog manager: success
I20260925 15:18:33.967927 2457 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a7db4ff52dca4ab08b2a471c0c94df52: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 15:18:34.168989 2523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:34.162540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37474 (local address 127.12.8.61:36113)
0925 15:18:34.163395 (+ 855us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:34.163401 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:34.163426 (+ 25us) server_negotiation.cc:407] Connection header received
0925 15:18:34.163494 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:34.163497 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:34.163632 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:34.163770 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:18:34.164254 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:34.165106 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:34.166495 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:34.167024 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:34.167161 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:34.167707 (+ 546us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:18:34.168289 (+ 582us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:18:34.168458 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:34.168585 (+ 127us) server_negotiation.cc:299] Negotiation successful
0925 15:18:34.168669 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":686,"thread_start_us":542,"threads_started":1}
I20260925 15:18:34.181936 2529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:34.175003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42784 (local address 127.12.8.60:46547)
0925 15:18:34.175540 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:34.175545 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:34.175863 (+ 318us) server_negotiation.cc:407] Connection header received
0925 15:18:34.176194 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:34.176198 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:34.176338 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:34.176465 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:18:34.176928 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:34.177585 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:34.179501 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:34.179991 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:34.180103 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:34.180691 (+ 588us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:18:34.181285 (+ 594us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:18:34.181407 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:34.181535 (+ 128us) server_negotiation.cc:299] Negotiation successful
0925 15:18:34.181601 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":268,"threads_started":1}
I20260925 15:18:34.202538 2523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:34.189341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37476 (local address 127.12.8.61:36113)
0925 15:18:34.189551 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:34.189557 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:34.195430 (+ 5873us) server_negotiation.cc:407] Connection header received
0925 15:18:34.195962 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:34.195968 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:34.196062 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:34.196122 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:18:34.196881 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:34.198033 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:34.200275 (+ 2242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:34.200974 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:34.201066 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:34.201684 (+ 618us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:18:34.201804 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:18:34.201868 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:34.202076 (+ 208us) server_negotiation.cc:299] Negotiation successful
0925 15:18:34.202161 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 15:18:34.202935 2529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:34.191957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42792 (local address 127.12.8.60:46547)
0925 15:18:34.195281 (+ 3324us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:34.195286 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:34.195688 (+ 402us) server_negotiation.cc:407] Connection header received
0925 15:18:34.195889 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:34.195896 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:34.195979 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:34.196061 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:18:34.196899 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:34.197995 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:34.200692 (+ 2697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:34.201322 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:34.201429 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:34.201949 (+ 520us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:18:34.202071 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:18:34.202158 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:34.202582 (+ 424us) server_negotiation.cc:299] Negotiation successful
0925 15:18:34.202658 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3204}
I20260925 15:18:34.204746 2535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:34.189509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58810 (local address 127.12.8.62:44539)
0925 15:18:34.196207 (+ 6698us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:34.196214 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:34.196235 (+ 21us) server_negotiation.cc:407] Connection header received
0925 15:18:34.196345 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:34.196352 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:34.196573 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:34.197360 (+ 787us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:18:34.197529 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:34.198618 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:34.200981 (+ 2363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:34.201620 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:34.202233 (+ 613us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:34.202417 (+ 184us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:18:34.203791 (+ 1374us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:18:34.203902 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:34.204405 (+ 503us) server_negotiation.cc:299] Negotiation successful
0925 15:18:34.204490 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6568,"thread_start_us":7247,"threads_started":1}
I20260925 15:18:34.214484 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 2175
I20260925 15:18:34.215567 2237 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:18:34.327350 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 2175
I20260925 15:18:34.348562 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 2249
I20260925 15:18:34.349633 2311 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:18:34.455225 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 2249
I20260925 15:18:34.475031 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 2323
I20260925 15:18:34.476151 2385 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:18:34.582171 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 2323
2026-09-25T15:18:34Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4983 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349162370577-12320-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:18:34 dist-test-slave-p468 krb5kdc[2543](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:18:34 dist-test-slave-p468 krb5kdc[2543](info): set up 2 sockets
Sep 25 15:18:34 dist-test-slave-p468 krb5kdc[2543](info): commencing operation
krb5kdc: starting...
W20260925 15:18:35.735307 12320 mini_kdc.cc:121] Time spent starting KDC: real 1.110s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349162370577-12320-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349162370577-12320-0/krb5kdc/test-user.keytab.
Sep 25 15:18:35 dist-test-slave-p468 krb5kdc[2543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349515, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:18:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:18:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.8.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab.
Entry for principal kudu/127.12.8.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab.
I20260925 15:18:35.989106 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349162370577-12320-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349162370577-12320-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349162370577-12320-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.62:45205
--webserver_interface=127.12.8.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab
--principal=kudu/127.12.8.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:36493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.62:45205,127.12.8.61:35345,127.12.8.60:44633
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:18:36.317847 2556 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 15:18:36.318265 2556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:18:36.318388 2556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:18:36.318542 2556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:18:36.325795 2556 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:18:36.325923 2556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:18:36.326023 2556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:18:36.326118 2556 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:18:36.326243 2556 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:18:36.337410 2556 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:36493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349162370577-12320-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349162370577-12320-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.8.62:45205,127.12.8.61:35345,127.12.8.60:44633
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.62.keytab
--principal=kudu/127.12.8.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.62:45205
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.8.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349162370577-12320-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:18:36.338861 2556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:18:36.340219 2556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:18:36.352128 2561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:36.352097 2564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:36.352106 2562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:18:36.352967 2556 server_base.cc:1034] running on GCE node
I20260925 15:18:36.353796 2556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:18:36.355235 2556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:18:36.356479 2556 hybrid_clock.cc:648] HybridClock initialized: now 1790349516356410 us; error 35 us; skew 500 ppm
I20260925 15:18:36.356901 2556 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:18:36 dist-test-slave-p468 krb5kdc[2543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349516, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:18:36.360219 2556 init.cc:377] Logged in from keytab as kudu/127.12.8.62@KRBTEST.COM (short username kudu)
I20260925 15:18:36.361740 2556 webserver.cc:460] Webserver started at http://127.12.8.62:33933/ using document root <none> and password file <none>
I20260925 15:18:36.362834 2556 fs_manager.cc:359] Metadata directory not provided
I20260925 15:18:36.363068 2556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:18:36.363389 2556 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:18:36.366364 2556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349162370577-12320-0/minicluster-data/master-0/data/instance:
uuid: "b4708b049c9b422f92dc7d46c01c4752"
format_stamp: "Formatted at 2026-09-25 15:18:36 on dist-test-slave-p468"
server_key: "ac5d4be88bad9559aae11b5d114d9dca"
server_key_iv: "172cf068d60790c06416645c41f68ff1"
server_key_version: "encryptionkey@0"
I20260925 15:18:36.367157 2556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349162370577-12320-0/minicluster-data/master-0/wal/instance:
uuid: "b4708b049c9b422f92dc7d46c01c4752"
format_stamp: "Formatted at 2026-09-25 15:18:36 on dist-test-slave-p468"
server_key: "ac5d4be88bad9559aae11b5d114d9dca"
server_key_iv: "172cf068d60790c06416645c41f68ff1"
server_key_version: "encryptionkey@0"
I20260925 15:18:36.372331 2556 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260925 15:18:36.375658 2571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:36.376966 2556 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 15:18:36.377159 2556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349162370577-12320-0/minicluster-data/master-0/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349162370577-12320-0/minicluster-data/master-0/wal
uuid: "b4708b049c9b422f92dc7d46c01c4752"
format_stamp: "Formatted at 2026-09-25 15:18:36 on dist-test-slave-p468"
server_key: "ac5d4be88bad9559aae11b5d114d9dca"
server_key_iv: "172cf068d60790c06416645c41f68ff1"
server_key_version: "encryptionkey@0"
I20260925 15:18:36.377375 2556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349162370577-12320-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349162370577-12320-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349162370577-12320-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:18:36.390669 2556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:18:36.394934 2556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:18:36.395205 2556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:18:36.414631 2556 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.62:45205
I20260925 15:18:36.414688 2622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.62:45205 every 8 connection(s)
I20260925 15:18:36.416083 2556 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349162370577-12320-0/minicluster-data/master-0/data/info.pb
I20260925 15:18:36.419811 2623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:18:36.424007 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 2556
I20260925 15:18:36.424203 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349162370577-12320-0/minicluster-data/master-0/wal/instance
I20260925 15:18:36.424731 12320 external_mini_cluster.cc:1428] Setting key 867761c2a187bf7380cb31773b67b7e0
I20260925 15:18:36.427721 2623 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.8.62" port: 45205 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:18:36 dist-test-slave-p468 krb5kdc[2543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349516, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.62@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.8.61@KRBTEST.COM; defaulting to no policy
I20260925 15:18:36.454329 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:36.431338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:45205 (local address 127.0.0.1:57744)
0925 15:18:36.432164 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:36.432212 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:36.434555 (+ 2343us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:36.435446 (+ 891us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:36.435456 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:36.436288 (+ 832us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:18:36.436777 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:36.436803 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:36.438483 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:36.438491 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:36.440354 (+ 1863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:36.440371 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:36.441410 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:36.441420 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:36.441673 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:36.442524 (+ 851us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:18:36.442559 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:18:36.445239 (+ 2680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:18:36.449793 (+ 4554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:36.449803 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:36.449812 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:36.450254 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:36.450746 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:36.450750 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:36.450752 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:36.450895 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:36.451899 (+ 1004us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:18:36.451904 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:18:36.452095 (+ 191us) client_negotiation.cc:769] Sending connection context
0925 15:18:36.452300 (+ 205us) client_negotiation.cc:240] Negotiation successful
0925 15:18:36.452774 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":373,"threads_started":1}
I20260925 15:18:36.454406 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:36.432625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57744 (local address 127.12.8.62:45205)
0925 15:18:36.433393 (+ 768us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:36.433403 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:36.433452 (+ 49us) server_negotiation.cc:407] Connection header received
0925 15:18:36.434954 (+ 1502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:36.434960 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:36.435197 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:36.435362 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:36.437014 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:36.438326 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:36.440528 (+ 2202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:36.441161 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:36.441311 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:36.445473 (+ 4162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:36.445484 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:36.445503 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:36.445549 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:36.449610 (+ 4061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:36.450399 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:36.450408 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:36.450412 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:36.450599 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:36.451018 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:36.451026 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:36.451030 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:36.451728 (+ 698us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:36.451958 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:36.452487 (+ 529us) server_negotiation.cc:299] Negotiation successful
0925 15:18:36.452903 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":137,"server-negotiator.queue_time_us":544,"thread_start_us":317,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.61@KRBTEST.COM" created.
I20260925 15:18:36.457357 2623 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.8.61" port: 35345 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:18:36.458520 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:36.458006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.61:35345 (local address 127.0.0.1:51246)
0925 15:18:36.458249 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:36.458373 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.61:35345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260925 15:18:36.458825 2574 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.8.61:35345: connect: Connection refused (error 111)
I20260925 15:18:36.460457 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:36.460016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.61:35345 (local address 127.0.0.1:51250)
0925 15:18:36.460184 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:36.460286 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.61:35345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260925 15:18:36.460693 2623 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.8.61:35345: Network error: Client connection negotiation failed: client connection to 127.12.8.61:35345: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.61.keytab.
Entry for principal kudu/127.12.8.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.61.keytab.
I20260925 15:18:36.485778 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349162370577-12320-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349162370577-12320-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349162370577-12320-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349162370577-12320-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.61:35345
--webserver_interface=127.12.8.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.61.keytab
--principal=kudu/127.12.8.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:36493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.62:45205,127.12.8.61:35345,127.12.8.60:44633
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:18:36.510043 2623 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.8.61" port: 35345 } attempt: 1
I20260925 15:18:36.511286 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:36.510826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.61:35345 (local address 127.0.0.1:51264)
0925 15:18:36.511029 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:36.511130 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.61:35345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260925 15:18:36.512369 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:36.511972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.61:35345 (local address 127.0.0.1:51276)
0925 15:18:36.512132 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:36.512214 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.61:35345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260925 15:18:36.512591 2623 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.8.61:35345: Network error: Client connection negotiation failed: client connection to 127.12.8.61:35345: connect: Connection refused (error 111)
I20260925 15:18:36.580870 2623 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.8.61" port: 35345 } attempt: 2
I20260925 15:18:36.582042 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:36.581550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.61:35345 (local address 127.0.0.1:51288)
0925 15:18:36.581744 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:36.581895 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.61:35345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260925 15:18:36.583112 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:36.582745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.61:35345 (local address 127.0.0.1:51298)
0925 15:18:36.582898 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:36.582984 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.61:35345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260925 15:18:36.583310 2623 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.8.61:35345: Network error: Client connection negotiation failed: client connection to 127.12.8.61:35345: connect: Connection refused (error 111)
I20260925 15:18:36.674594 2623 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.8.61" port: 35345 } attempt: 3
I20260925 15:18:36.675760 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:36.675311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.61:35345 (local address 127.0.0.1:51312)
0925 15:18:36.675522 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:36.675622 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.61:35345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260925 15:18:36.676771 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:36.676431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.61:35345 (local address 127.0.0.1:51318)
0925 15:18:36.676570 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:36.676645 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.61:35345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260925 15:18:36.677055 2623 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.8.61:35345: Network error: Client connection negotiation failed: client connection to 127.12.8.61:35345: connect: Connection refused (error 111)
W20260925 15:18:36.811873 2630 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 15:18:36.812223 2630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:18:36.812345 2630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:18:36.812492 2630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:18:36.819837 2630 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:18:36.819927 2630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:18:36.820012 2630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:18:36.820107 2630 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:18:36.820216 2630 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:18:36.820338 2623 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.8.61" port: 35345 } attempt: 4
I20260925 15:18:36.821578 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:36.821016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.61:35345 (local address 127.0.0.1:51334)
0925 15:18:36.821349 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:36.821451 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.61:35345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197}
I20260925 15:18:36.822638 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:36.822302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.61:35345 (local address 127.0.0.1:51350)
0925 15:18:36.822449 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:36.822524 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.61:35345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260925 15:18:36.822923 2623 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.8.61:35345: Network error: Client connection negotiation failed: client connection to 127.12.8.61:35345: connect: Connection refused (error 111)
I20260925 15:18:36.830799 2630 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:36493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349162370577-12320-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349162370577-12320-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.12.8.62:45205,127.12.8.61:35345,127.12.8.60:44633
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.61.keytab
--principal=kudu/127.12.8.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.61:35345
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349162370577-12320-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.12.8.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349162370577-12320-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:18:36.832093 2630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:18:36.833374 2630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:18:36.845515 2635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:36.845489 2638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:36.845547 2636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:18:36.846357 2630 server_base.cc:1034] running on GCE node
I20260925 15:18:36.847126 2630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:18:36.848464 2630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:18:36.849717 2630 hybrid_clock.cc:648] HybridClock initialized: now 1790349516849652 us; error 42 us; skew 500 ppm
I20260925 15:18:36.850062 2630 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:18:36 dist-test-slave-p468 krb5kdc[2543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349516, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:18:36.853209 2630 init.cc:377] Logged in from keytab as kudu/127.12.8.61@KRBTEST.COM (short username kudu)
I20260925 15:18:36.854629 2630 webserver.cc:460] Webserver started at http://127.12.8.61:40173/ using document root <none> and password file <none>
I20260925 15:18:36.855250 2630 fs_manager.cc:359] Metadata directory not provided
I20260925 15:18:36.855336 2630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:18:36.855589 2630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:18:36.858825 2630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349162370577-12320-0/minicluster-data/master-1/data/instance:
uuid: "28daa46279c7450883d7a210d6c0b2b8"
format_stamp: "Formatted at 2026-09-25 15:18:36 on dist-test-slave-p468"
server_key: "235cec671e1f3bb5f0d1c04f17e60ec4"
server_key_iv: "50573a6c60c3b5234f8550203d71b79e"
server_key_version: "encryptionkey@0"
I20260925 15:18:36.859965 2630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349162370577-12320-0/minicluster-data/master-1/wal/instance:
uuid: "28daa46279c7450883d7a210d6c0b2b8"
format_stamp: "Formatted at 2026-09-25 15:18:36 on dist-test-slave-p468"
server_key: "235cec671e1f3bb5f0d1c04f17e60ec4"
server_key_iv: "50573a6c60c3b5234f8550203d71b79e"
server_key_version: "encryptionkey@0"
I20260925 15:18:36.867616 2630 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260925 15:18:36.872644 2645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:36.874504 2630 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 15:18:36.874702 2630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349162370577-12320-0/minicluster-data/master-1/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349162370577-12320-0/minicluster-data/master-1/wal
uuid: "28daa46279c7450883d7a210d6c0b2b8"
format_stamp: "Formatted at 2026-09-25 15:18:36 on dist-test-slave-p468"
server_key: "235cec671e1f3bb5f0d1c04f17e60ec4"
server_key_iv: "50573a6c60c3b5234f8550203d71b79e"
server_key_version: "encryptionkey@0"
I20260925 15:18:36.874984 2630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349162370577-12320-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349162370577-12320-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349162370577-12320-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:18:36.889658 2630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:18:36.893592 2630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:18:36.893847 2630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:18:36.912542 2630 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.61:35345
I20260925 15:18:36.912573 2696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.61:35345 every 8 connection(s)
I20260925 15:18:36.914110 2630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349162370577-12320-0/minicluster-data/master-1/data/info.pb
I20260925 15:18:36.918046 2697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:18:36.923309 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 2630
I20260925 15:18:36.923506 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349162370577-12320-0/minicluster-data/master-1/wal/instance
I20260925 15:18:36.923935 12320 external_mini_cluster.cc:1428] Setting key 0976c64d3435119fdafbea653dcc24ee
I20260925 15:18:36.924551 2697 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.8.62" port: 45205 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:18:36 dist-test-slave-p468 krb5kdc[2543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349516, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.61@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
I20260925 15:18:36.945715 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:36.927183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57760 (local address 127.12.8.62:45205)
0925 15:18:36.927360 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:36.927364 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:36.928533 (+ 1169us) server_negotiation.cc:407] Connection header received
0925 15:18:36.930417 (+ 1884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:36.930423 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:36.930516 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:36.930630 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:36.932131 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:36.932809 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:36.935271 (+ 2462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:36.935678 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:36.935788 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:36.939834 (+ 4046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:36.939839 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:36.939842 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:36.939877 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:36.942577 (+ 2700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:36.943328 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:36.943336 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:36.943345 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:36.943462 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:36.943877 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:36.943882 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:36.943885 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:36.944173 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:36.944350 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:36.945305 (+ 955us) server_negotiation.cc:299] Negotiation successful
0925 15:18:36.945482 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 15:18:36.947382 2699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:36.927629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:45205 (local address 127.0.0.1:57760)
0925 15:18:36.928374 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:36.928411 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:36.930092 (+ 1681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:36.930900 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:36.930908 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:36.931421 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:18:36.931953 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:36.931969 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:36.932969 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:36.932973 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:36.935113 (+ 2140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:36.935130 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:36.935912 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:36.935922 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:36.936237 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:36.936974 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:18:36.937010 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:18:36.939580 (+ 2570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:18:36.942727 (+ 3147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:36.942737 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:36.942755 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:36.943193 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:36.943580 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:36.943584 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:36.943588 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:36.943759 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:36.944349 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:18:36.944362 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:18:36.944882 (+ 520us) client_negotiation.cc:769] Sending connection context
0925 15:18:36.945226 (+ 344us) client_negotiation.cc:240] Negotiation successful
0925 15:18:36.945849 (+ 623us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":327,"threads_started":1}
WARNING: no policy specified for kudu/127.12.8.60@KRBTEST.COM; defaulting to no policy
I20260925 15:18:36.949210 2697 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.8.61" port: 35345 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.8.60@KRBTEST.COM" created.
Sep 25 15:18:36 dist-test-slave-p468 krb5kdc[2543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349516, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.61@KRBTEST.COM for kudu/127.12.8.61@KRBTEST.COM
I20260925 15:18:36.965186 2701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:36.950276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51362 (local address 127.12.8.61:35345)
0925 15:18:36.950869 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:36.950875 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:36.950899 (+ 24us) server_negotiation.cc:407] Connection header received
0925 15:18:36.950939 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:36.950945 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:36.951113 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:36.951294 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:36.952218 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:36.953283 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:36.955086 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:36.955608 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:36.955707 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:36.959362 (+ 3655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:36.959377 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:36.959396 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:36.959443 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:36.962380 (+ 2937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:36.963170 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:36.963176 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:36.963178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:36.963294 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:36.963700 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:36.963703 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:36.963704 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:36.964004 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:36.964153 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:36.964607 (+ 454us) server_negotiation.cc:299] Negotiation successful
0925 15:18:36.964904 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":110,"server-negotiator.queue_time_us":462,"thread_start_us":368,"threads_started":1}
I20260925 15:18:36.965265 2699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:36.950011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.61:35345 (local address 127.0.0.1:51362)
0925 15:18:36.950158 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:36.950174 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:36.950345 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:36.951322 (+ 977us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:36.951328 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:36.951804 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:18:36.952022 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:36.952038 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:36.953438 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:36.953446 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:36.954937 (+ 1491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:36.954953 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:36.955782 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:36.955793 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:36.955997 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:36.956581 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:18:36.956607 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:18:36.959144 (+ 2537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:18:36.962498 (+ 3354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:36.962505 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:36.962513 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:36.963046 (+ 533us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:36.963404 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:36.963411 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:36.963418 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:36.963595 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:36.964144 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:18:36.964152 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:18:36.964321 (+ 169us) client_negotiation.cc:769] Sending connection context
0925 15:18:36.964473 (+ 152us) client_negotiation.cc:240] Negotiation successful
0925 15:18:36.964792 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260925 15:18:36.967767 2697 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.8.60" port: 44633 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:18:36.968758 2699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:36.968374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.60:44633 (local address 127.0.0.1:36874)
0925 15:18:36.968529 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:36.968636 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.60:44633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260925 15:18:36.969067 2648 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.8.60:44633: connect: Connection refused (error 111)
I20260925 15:18:36.970590 2699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:36.970304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.60:44633 (local address 127.0.0.1:36888)
0925 15:18:36.970426 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:36.970501 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.60:44633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260925 15:18:36.970880 2697 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.8.60:44633: Network error: Client connection negotiation failed: client connection to 127.12.8.60:44633: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.8.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.60.keytab.
Entry for principal kudu/127.12.8.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.60.keytab.
I20260925 15:18:36.985296 12320 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBtce3V/build/asan/bin/kudu
/tmp/dist-test-taskBtce3V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349162370577-12320-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349162370577-12320-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349162370577-12320-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349162370577-12320-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.60:44633
--webserver_interface=127.12.8.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.60.keytab
--principal=kudu/127.12.8.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.8.20:36493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.8.62:45205,127.12.8.61:35345,127.12.8.60:44633
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349162370577-12320-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349162370577-12320-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349162370577-12320-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:18:37.020205 2697 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.8.60" port: 44633 } attempt: 1
I20260925 15:18:37.021406 2699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:37.020958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.60:44633 (local address 127.0.0.1:36900)
0925 15:18:37.021149 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:37.021238 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.60:44633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260925 15:18:37.022401 2699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:37.022013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.60:44633 (local address 127.0.0.1:36902)
0925 15:18:37.022154 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:37.022298 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.60:44633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 15:18:37.022600 2697 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.8.60:44633: Network error: Client connection negotiation failed: client connection to 127.12.8.60:44633: connect: Connection refused (error 111)
I20260925 15:18:37.090829 2697 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.8.60" port: 44633 } attempt: 2
I20260925 15:18:37.092012 2699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:37.091539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.60:44633 (local address 127.0.0.1:36916)
0925 15:18:37.091739 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:37.091848 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.60:44633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260925 15:18:37.093029 2699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:37.092653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.60:44633 (local address 127.0.0.1:36930)
0925 15:18:37.092845 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:37.092921 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.60:44633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
W20260925 15:18:37.093241 2697 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.8.60:44633: Network error: Client connection negotiation failed: client connection to 127.12.8.60:44633: connect: Connection refused (error 111)
I20260925 15:18:37.122262 2623 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.8.61" port: 35345 } attempt: 5
Sep 25 15:18:37 dist-test-slave-p468 krb5kdc[2543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349516, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.62@KRBTEST.COM for kudu/127.12.8.61@KRBTEST.COM
I20260925 15:18:37.136880 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:37.123149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.61:35345 (local address 127.0.0.1:51368)
0925 15:18:37.123377 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:37.123398 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:37.123581 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:37.124072 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:37.124077 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:37.124496 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:18:37.124675 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:37.124684 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:37.125691 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:37.125698 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:37.126634 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:37.126646 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:37.127407 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:37.127413 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:37.127594 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:37.128261 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:18:37.128279 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:18:37.130656 (+ 2377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:18:37.134516 (+ 3860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:37.134520 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:37.134523 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:37.134927 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:37.135311 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:37.135314 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:37.135316 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:37.135441 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:37.136151 (+ 710us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:18:37.136155 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:18:37.136302 (+ 147us) client_negotiation.cc:769] Sending connection context
0925 15:18:37.136417 (+ 115us) client_negotiation.cc:240] Negotiation successful
0925 15:18:37.136629 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79}
I20260925 15:18:37.137188 2701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:37.123234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51368 (local address 127.12.8.61:35345)
0925 15:18:37.123410 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:37.123417 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:37.123514 (+ 97us) server_negotiation.cc:407] Connection header received
0925 15:18:37.123811 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:37.123817 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:37.123911 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:37.124033 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:37.124838 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:37.125547 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:37.126787 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:37.127255 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:37.127401 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:37.130862 (+ 3461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:37.130868 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:37.130872 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:37.130914 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:37.134388 (+ 3474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:37.135043 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:37.135049 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:37.135052 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:37.135198 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:37.135549 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:37.135555 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:37.135561 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:37.135999 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:37.136179 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:37.136586 (+ 407us) server_negotiation.cc:299] Negotiation successful
0925 15:18:37.136875 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 15:18:37.138290 2623 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.8.60" port: 44633 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:18:37.139298 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:37.138930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.60:44633 (local address 127.0.0.1:36934)
0925 15:18:37.139082 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:37.139153 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.60:44633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260925 15:18:37.140223 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:37.139919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.60:44633 (local address 127.0.0.1:36942)
0925 15:18:37.140059 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:37.140124 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.60:44633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 15:18:37.140436 2623 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.8.60:44633: Network error: Client connection negotiation failed: client connection to 127.12.8.60:44633: connect: Connection refused (error 111)
I20260925 15:18:37.184499 2697 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.8.60" port: 44633 } attempt: 3
I20260925 15:18:37.185837 2699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:37.185320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.60:44633 (local address 127.0.0.1:36946)
0925 15:18:37.185551 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:37.185678 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.60:44633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260925 15:18:37.186941 2699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:37.186527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.60:44633 (local address 127.0.0.1:36954)
0925 15:18:37.186747 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:37.186824 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.60:44633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
W20260925 15:18:37.187150 2697 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.8.60:44633: Network error: Client connection negotiation failed: client connection to 127.12.8.60:44633: connect: Connection refused (error 111)
I20260925 15:18:37.191635 2623 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.8.60" port: 44633 } attempt: 1
I20260925 15:18:37.192545 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:37.192159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.60:44633 (local address 127.0.0.1:36966)
0925 15:18:37.192328 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:37.192409 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.60:44633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260925 15:18:37.193459 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:37.193159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.60:44633 (local address 127.0.0.1:36980)
0925 15:18:37.193300 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:37.193366 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.60:44633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 15:18:37.193665 2623 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.8.60:44633: Network error: Client connection negotiation failed: client connection to 127.12.8.60:44633: connect: Connection refused (error 111)
I20260925 15:18:37.261873 2623 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.8.60" port: 44633 } attempt: 2
I20260925 15:18:37.263000 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:37.262547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.60:44633 (local address 127.0.0.1:36996)
0925 15:18:37.262738 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:37.262850 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.60:44633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260925 15:18:37.264039 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:37.263724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.60:44633 (local address 127.0.0.1:37002)
0925 15:18:37.263883 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:37.263945 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.60:44633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260925 15:18:37.264329 2623 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.8.60:44633: Network error: Client connection negotiation failed: client connection to 127.12.8.60:44633: connect: Connection refused (error 111)
W20260925 15:18:37.310055 2704 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 15:18:37.310441 2704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:18:37.310564 2704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:18:37.310681 2704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:18:37.317956 2704 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:18:37.318035 2704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:18:37.318127 2704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:18:37.318233 2704 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:18:37.318338 2704 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:18:37.328122 2704 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.8.20:36493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349162370577-12320-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349162370577-12320-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.12.8.62:45205,127.12.8.61:35345,127.12.8.60:44633
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349162370577-12320-0/krb5kdc/kudu_127.12.8.60.keytab
--principal=kudu/127.12.8.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.8.60:44633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349162370577-12320-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.12.8.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349162370577-12320-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:18:37.329399 2704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:18:37.330366 2697 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.8.60" port: 44633 } attempt: 4
I20260925 15:18:37.330777 2704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 15:18:37.331851 2699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:37.331172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.60:44633 (local address 127.0.0.1:37018)
0925 15:18:37.331510 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:37.331634 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.60:44633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260925 15:18:37.333304 2699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:37.332870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.60:44633 (local address 127.0.0.1:37020)
0925 15:18:37.333068 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:37.333155 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.60:44633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260925 15:18:37.334255 2697 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.8.60:44633: Network error: Client connection negotiation failed: client connection to 127.12.8.60:44633: connect: Connection refused (error 111)
W20260925 15:18:37.343428 2710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:37.343488 2712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:18:37.343516 2709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:18:37.344396 2704 server_base.cc:1034] running on GCE node
I20260925 15:18:37.345227 2704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:18:37.346715 2704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:18:37.347954 2704 hybrid_clock.cc:648] HybridClock initialized: now 1790349517347882 us; error 58 us; skew 500 ppm
I20260925 15:18:37.348276 2704 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:18:37 dist-test-slave-p468 krb5kdc[2543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349517, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:18:37.351365 2704 init.cc:377] Logged in from keytab as kudu/127.12.8.60@KRBTEST.COM (short username kudu)
I20260925 15:18:37.352880 2704 webserver.cc:460] Webserver started at http://127.12.8.60:32883/ using document root <none> and password file <none>
I20260925 15:18:37.353571 2704 fs_manager.cc:359] Metadata directory not provided
I20260925 15:18:37.353700 2704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:18:37.354022 2704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:18:37.357002 2704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349162370577-12320-0/minicluster-data/master-2/data/instance:
uuid: "96f1a12b1f0b400087fed6e52f0b03d8"
format_stamp: "Formatted at 2026-09-25 15:18:37 on dist-test-slave-p468"
server_key: "f238ec88a9b562e272bccea0c27b1e4a"
server_key_iv: "f9fde97b19228aa56bb37de7084c2947"
server_key_version: "encryptionkey@0"
I20260925 15:18:37.357836 2704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349162370577-12320-0/minicluster-data/master-2/wal/instance:
uuid: "96f1a12b1f0b400087fed6e52f0b03d8"
format_stamp: "Formatted at 2026-09-25 15:18:37 on dist-test-slave-p468"
server_key: "f238ec88a9b562e272bccea0c27b1e4a"
server_key_iv: "f9fde97b19228aa56bb37de7084c2947"
server_key_version: "encryptionkey@0"
I20260925 15:18:37.363152 2704 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260925 15:18:37.366530 2719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:18:37.367849 2704 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 15:18:37.368026 2704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349162370577-12320-0/minicluster-data/master-2/data,/tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349162370577-12320-0/minicluster-data/master-2/wal
uuid: "96f1a12b1f0b400087fed6e52f0b03d8"
format_stamp: "Formatted at 2026-09-25 15:18:37 on dist-test-slave-p468"
server_key: "f238ec88a9b562e272bccea0c27b1e4a"
server_key_iv: "f9fde97b19228aa56bb37de7084c2947"
server_key_version: "encryptionkey@0"
I20260925 15:18:37.368244 2704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349162370577-12320-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349162370577-12320-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349162370577-12320-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:18:37.370530 2623 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.8.60" port: 44633 } attempt: 3
I20260925 15:18:37.371676 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:37.371154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.60:44633 (local address 127.0.0.1:37036)
0925 15:18:37.371382 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:37.371495 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.60:44633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260925 15:18:37.372608 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:37.372284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.60:44633 (local address 127.0.0.1:37052)
0925 15:18:37.372424 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:37.372514 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.8.60:44633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 15:18:37.372816 2623 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.8.60:44633: Network error: Client connection negotiation failed: client connection to 127.12.8.60:44633: connect: Connection refused (error 111)
I20260925 15:18:37.386834 2704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:18:37.391000 2704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:18:37.391269 2704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:18:37.409686 2704 rpc_server.cc:307] RPC server started. Bound to: 127.12.8.60:44633
I20260925 15:18:37.409693 2770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.8.60:44633 every 8 connection(s)
I20260925 15:18:37.411070 2704 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349162370577-12320-0/minicluster-data/master-2/data/info.pb
I20260925 15:18:37.414738 2771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:18:37.418733 12320 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBtce3V/build/asan/bin/kudu as pid 2704
I20260925 15:18:37.418921 12320 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBtce3V/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349162370577-12320-0/minicluster-data/master-2/wal/instance
I20260925 15:18:37.419459 12320 external_mini_cluster.cc:1428] Setting key d812c6a2839f48c85896e48ae8513460
I20260925 15:18:37.422978 2771 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.8.62" port: 45205 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:18:37 dist-test-slave-p468 krb5kdc[2543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349517, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.60@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
Sep 25 15:18:37 dist-test-slave-p468 krb5kdc[2543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349515, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.60@KRBTEST.COM
I20260925 15:18:37.445062 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:37.426305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57764 (local address 127.12.8.62:45205)
0925 15:18:37.426468 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:37.426472 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:37.427217 (+ 745us) server_negotiation.cc:407] Connection header received
0925 15:18:37.427950 (+ 733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:37.427955 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:37.428035 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:37.428140 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:37.429835 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:37.431193 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:37.433219 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:37.433850 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:37.433988 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:37.438317 (+ 4329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:37.438327 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:37.438334 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:37.438381 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:37.441308 (+ 2927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:37.442386 (+ 1078us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:37.442390 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:37.442392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:37.442522 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:37.443051 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:37.443054 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:37.443056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:37.443425 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:37.443584 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:37.444582 (+ 998us) server_negotiation.cc:299] Negotiation successful
0925 15:18:37.444754 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 15:18:37.446415 2775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:37.426316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.62:45205 (local address 127.0.0.1:57764)
0925 15:18:37.427078 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:37.427118 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:37.427713 (+ 595us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:37.428210 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:37.428217 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:37.429088 (+ 871us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:18:37.429625 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:37.429651 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:37.431357 (+ 1706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:37.431361 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:37.433067 (+ 1706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:37.433080 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:37.434079 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:37.434084 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:37.434298 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:37.435072 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:18:37.435113 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:18:37.438104 (+ 2991us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:18:37.441445 (+ 3341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:37.441463 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:37.441477 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:37.442245 (+ 768us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:37.442646 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:37.442652 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:37.442657 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:37.442915 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:37.443555 (+ 640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:18:37.443561 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:18:37.444091 (+ 530us) client_negotiation.cc:769] Sending connection context
0925 15:18:37.444426 (+ 335us) client_negotiation.cc:240] Negotiation successful
0925 15:18:37.444974 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"mutex_wait_us":38,"spinlock_wait_cycles":25984,"thread_start_us":359,"threads_started":1}
I20260925 15:18:37.446682 2774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:37.421260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37062 (local address 127.12.8.60:44633)
0925 15:18:37.424396 (+ 3136us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:37.424412 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:37.424810 (+ 398us) server_negotiation.cc:407] Connection header received
0925 15:18:37.427697 (+ 2887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:37.427718 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:37.427952 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:37.428169 (+ 217us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:37.430047 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:37.431823 (+ 1776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:37.433501 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:37.434412 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:37.434517 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:37.439189 (+ 4672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:37.439202 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:37.439222 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:37.439270 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:37.443152 (+ 3882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:37.443957 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:37.443965 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:37.443969 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:37.444147 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:37.444542 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:37.444547 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:37.444550 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:37.444976 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:37.445297 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:37.445902 (+ 605us) server_negotiation.cc:299] Negotiation successful
0925 15:18:37.446161 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2836,"thread_start_us":2558,"threads_started":1}
I20260925 15:18:37.448120 2771 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.8.61" port: 35345 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:18:37 dist-test-slave-p468 krb5kdc[2543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349517, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.60@KRBTEST.COM for kudu/127.12.8.61@KRBTEST.COM
Sep 25 15:18:37 dist-test-slave-p468 krb5kdc[2543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349515, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.62@KRBTEST.COM
I20260925 15:18:37.461163 2775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:37.449074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.61:35345 (local address 127.0.0.1:51370)
0925 15:18:37.449281 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:37.449301 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:37.449405 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:37.449751 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:37.449755 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:37.450108 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:18:37.450330 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:37.450339 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:37.451292 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:37.451298 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:37.452181 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:37.452192 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:37.452893 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:37.452899 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:37.453091 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:37.453675 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:18:37.453693 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:18:37.455686 (+ 1993us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:18:37.458789 (+ 3103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:37.458795 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:37.458799 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:37.459152 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:37.459527 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:37.459532 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:37.459536 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:37.459692 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:37.460227 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:18:37.460232 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:18:37.460373 (+ 141us) client_negotiation.cc:769] Sending connection context
0925 15:18:37.460513 (+ 140us) client_negotiation.cc:240] Negotiation successful
0925 15:18:37.460804 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260925 15:18:37.461220 2701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:37.449292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51370 (local address 127.12.8.61:35345)
0925 15:18:37.449455 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:37.449459 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:37.449474 (+ 15us) server_negotiation.cc:407] Connection header received
0925 15:18:37.449524 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:37.449527 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:37.449583 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:37.449738 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:37.450477 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:37.451146 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:37.452336 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:37.452747 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:37.452875 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:37.455866 (+ 2991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:37.455871 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:37.455874 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:37.455906 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:37.458661 (+ 2755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:37.459271 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:37.459278 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:37.459281 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:37.459410 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:37.459824 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:37.459830 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:37.459834 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:37.460100 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:37.460266 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:37.460600 (+ 334us) server_negotiation.cc:299] Negotiation successful
0925 15:18:37.460804 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 15:18:37.462144 2771 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.8.60" port: 44633 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:18:37.463603 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:37.449681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57778 (local address 127.12.8.62:45205)
0925 15:18:37.449831 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:37.449836 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:37.449853 (+ 17us) server_negotiation.cc:407] Connection header received
0925 15:18:37.449933 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:37.449938 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:37.450007 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:37.450105 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:37.451237 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:37.451953 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:37.453103 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:37.453707 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:37.453856 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:37.458043 (+ 4187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:37.458049 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:37.458054 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:37.458092 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:37.460710 (+ 2618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:37.461405 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:37.461413 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:37.461418 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:37.461553 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:37.461994 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:37.462000 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:37.462003 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:37.462379 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:37.462677 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:37.463027 (+ 350us) server_negotiation.cc:299] Negotiation successful
0925 15:18:37.463254 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
Sep 25 15:18:37 dist-test-slave-p468 krb5kdc[2543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349517, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.60@KRBTEST.COM for kudu/127.12.8.60@KRBTEST.COM
Sep 25 15:18:37 dist-test-slave-p468 krb5kdc[2543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349515, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.8.61@KRBTEST.COM
I20260925 15:18:37.476168 2774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:37.462916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37070 (local address 127.12.8.60:44633)
0925 15:18:37.463075 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:37.463080 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:37.463206 (+ 126us) server_negotiation.cc:407] Connection header received
0925 15:18:37.463365 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:37.463370 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:37.463441 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:37.463555 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:37.464264 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:37.464925 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:37.466697 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:37.467160 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:37.467292 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:37.470824 (+ 3532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:37.470829 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:37.470832 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:37.470860 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:37.473420 (+ 2560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:37.474069 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:37.474076 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:37.474081 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:37.474258 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:37.474657 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:37.474662 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:37.474665 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:37.475035 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:37.475179 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:37.475588 (+ 409us) server_negotiation.cc:299] Negotiation successful
0925 15:18:37.475766 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 15:18:37.476419 2775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:37.462849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.60:44633 (local address 127.0.0.1:37070)
0925 15:18:37.463114 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:37.463131 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:37.463237 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:37.463571 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:37.463575 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:37.464002 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:18:37.464124 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:37.464134 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:37.465116 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:37.465123 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:37.466549 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:37.466566 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:37.467308 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:37.467312 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:37.467484 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:37.468033 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:18:37.468050 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:18:37.470546 (+ 2496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:18:37.473551 (+ 3005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:37.473558 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:37.473563 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:37.473944 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:37.474404 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:37.474407 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:37.474410 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:37.474545 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:37.475173 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:18:37.475178 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:18:37.475334 (+ 156us) client_negotiation.cc:769] Sending connection context
0925 15:18:37.475508 (+ 174us) client_negotiation.cc:240] Negotiation successful
0925 15:18:37.475953 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":127}
I20260925 15:18:37.481792 2701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:37.465158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51382 (local address 127.12.8.61:35345)
0925 15:18:37.465309 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:37.465312 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:37.465327 (+ 15us) server_negotiation.cc:407] Connection header received
0925 15:18:37.465484 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:37.465487 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:37.465539 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:37.465612 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:37.467146 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:37.468099 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:37.469794 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:37.470325 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:37.470420 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:37.475213 (+ 4793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:37.475219 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:37.475223 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:37.475262 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:37.478670 (+ 3408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:37.479480 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:37.479484 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:37.479486 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:37.479634 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:37.480088 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:37.480095 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:37.480097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:37.480479 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:37.480790 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:37.481104 (+ 314us) server_negotiation.cc:299] Negotiation successful
0925 15:18:37.481345 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 15:18:37.482349 2771 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96f1a12b1f0b400087fed6e52f0b03d8: Bootstrap starting.
I20260925 15:18:37.485922 2771 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 96f1a12b1f0b400087fed6e52f0b03d8: Neither blocks nor log segments found. Creating new log.
I20260925 15:18:37.487172 2771 log.cc:824] T 00000000000000000000000000000000 P 96f1a12b1f0b400087fed6e52f0b03d8: Log is configured to *not* fsync() on all Append() calls
I20260925 15:18:37.488627 12320 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:18:37.489894 2771 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96f1a12b1f0b400087fed6e52f0b03d8: No bootstrap required, opened a new log
I20260925 15:18:37.491304 2598 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "b4708b049c9b422f92dc7d46c01c4752"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57778
I20260925 15:18:37.497293 2771 raft_consensus.cc:348] T 00000000000000000000000000000000 P 96f1a12b1f0b400087fed6e52f0b03d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4708b049c9b422f92dc7d46c01c4752" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 45205 } } peers { permanent_uuid: "28daa46279c7450883d7a210d6c0b2b8" member_type: VOTER last_known_addr { host: "127.12.8.61" port: 35345 } } peers { permanent_uuid: "96f1a12b1f0b400087fed6e52f0b03d8" member_type: VOTER last_known_addr { host: "127.12.8.60" port: 44633 } }
I20260925 15:18:37.497659 2771 raft_consensus.cc:729] T 00000000000000000000000000000000 P 96f1a12b1f0b400087fed6e52f0b03d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96f1a12b1f0b400087fed6e52f0b03d8, State: Initialized, Role: FOLLOWER
I20260925 15:18:37.498132 2771 consensus_queue.cc:260] T 00000000000000000000000000000000 P 96f1a12b1f0b400087fed6e52f0b03d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4708b049c9b422f92dc7d46c01c4752" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 45205 } } peers { permanent_uuid: "28daa46279c7450883d7a210d6c0b2b8" member_type: VOTER last_known_addr { host: "127.12.8.61" port: 35345 } } peers { permanent_uuid: "96f1a12b1f0b400087fed6e52f0b03d8" member_type: VOTER last_known_addr { host: "127.12.8.60" port: 44633 } }
I20260925 15:18:37.499154 2777 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96f1a12b1f0b400087fed6e52f0b03d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4708b049c9b422f92dc7d46c01c4752" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 45205 } } peers { permanent_uuid: "28daa46279c7450883d7a210d6c0b2b8" member_type: VOTER last_known_addr { host: "127.12.8.61" port: 35345 } } peers { permanent_uuid: "96f1a12b1f0b400087fed6e52f0b03d8" member_type: VOTER last_known_addr { host: "127.12.8.60" port: 44633 } } }
I20260925 15:18:37.499538 2777 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96f1a12b1f0b400087fed6e52f0b03d8 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:18:37.500265 2771 sys_catalog.cc:564] T 00000000000000000000000000000000 P 96f1a12b1f0b400087fed6e52f0b03d8 [sys.catalog]: configured and running, proceeding with master startup.
W20260925 15:18:37.509756 2788 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 96f1a12b1f0b400087fed6e52f0b03d8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:18:37.509914 2788 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:18:37.507562 2704 master_runner.cc:186] Error getting master registration for 127.12.8.62:45205: OK, instance_id { permanent_uuid: "b4708b049c9b422f92dc7d46c01c4752" instance_seqno: 1790349516409175 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 15:18:37.518872 2704 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 15:18:37.549988 2623 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.8.60" port: 44633 } attempt: 4
Sep 25 15:18:37 dist-test-slave-p468 krb5kdc[2543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349516, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.62@KRBTEST.COM for kudu/127.12.8.60@KRBTEST.COM
I20260925 15:18:37.566923 2774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:37.550942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37080 (local address 127.12.8.60:44633)
0925 15:18:37.551116 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:37.551120 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:37.551251 (+ 131us) server_negotiation.cc:407] Connection header received
0925 15:18:37.551474 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:37.551479 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:37.551581 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:37.551682 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:37.552667 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:37.553805 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:37.555612 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:37.556375 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:37.556498 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:37.560368 (+ 3870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:37.560374 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:37.560379 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:37.560422 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:37.564167 (+ 3745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:37.564788 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:37.564796 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:37.564800 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:37.564931 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:37.565281 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:37.565287 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:37.565293 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:37.565725 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:37.565881 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:37.566299 (+ 418us) server_negotiation.cc:299] Negotiation successful
0925 15:18:37.566505 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 15:18:37.566913 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:37.550850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.60:44633 (local address 127.0.0.1:37080)
0925 15:18:37.551144 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:37.551165 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:37.551313 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:37.551759 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:37.551765 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:37.552262 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:18:37.552518 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:37.552531 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:37.553964 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:37.553970 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:37.555470 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:37.555486 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:37.556536 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:37.556542 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:37.556751 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:37.557433 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:18:37.557457 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:18:37.560173 (+ 2716us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:18:37.564296 (+ 4123us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:37.564302 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:37.564306 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:37.564658 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:37.565028 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:37.565034 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:37.565038 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:37.565174 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:37.565861 (+ 687us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:18:37.565865 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:18:37.566030 (+ 165us) client_negotiation.cc:769] Sending connection context
0925 15:18:37.566170 (+ 140us) client_negotiation.cc:240] Negotiation successful
0925 15:18:37.566497 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171}
I20260925 15:18:37.573176 2623 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4708b049c9b422f92dc7d46c01c4752: Bootstrap starting.
I20260925 15:18:37.577217 2623 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b4708b049c9b422f92dc7d46c01c4752: Neither blocks nor log segments found. Creating new log.
I20260925 15:18:37.578573 2623 log.cc:824] T 00000000000000000000000000000000 P b4708b049c9b422f92dc7d46c01c4752: Log is configured to *not* fsync() on all Append() calls
I20260925 15:18:37.582208 2623 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4708b049c9b422f92dc7d46c01c4752: No bootstrap required, opened a new log
I20260925 15:18:37.589946 2623 raft_consensus.cc:348] T 00000000000000000000000000000000 P b4708b049c9b422f92dc7d46c01c4752 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4708b049c9b422f92dc7d46c01c4752" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 45205 } } peers { permanent_uuid: "28daa46279c7450883d7a210d6c0b2b8" member_type: VOTER last_known_addr { host: "127.12.8.61" port: 35345 } } peers { permanent_uuid: "96f1a12b1f0b400087fed6e52f0b03d8" member_type: VOTER last_known_addr { host: "127.12.8.60" port: 44633 } }
I20260925 15:18:37.590358 2623 raft_consensus.cc:729] T 00000000000000000000000000000000 P b4708b049c9b422f92dc7d46c01c4752 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4708b049c9b422f92dc7d46c01c4752, State: Initialized, Role: FOLLOWER
I20260925 15:18:37.590874 2623 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4708b049c9b422f92dc7d46c01c4752 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4708b049c9b422f92dc7d46c01c4752" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 45205 } } peers { permanent_uuid: "28daa46279c7450883d7a210d6c0b2b8" member_type: VOTER last_known_addr { host: "127.12.8.61" port: 35345 } } peers { permanent_uuid: "96f1a12b1f0b400087fed6e52f0b03d8" member_type: VOTER last_known_addr { host: "127.12.8.60" port: 44633 } }
I20260925 15:18:37.591864 2791 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4708b049c9b422f92dc7d46c01c4752 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4708b049c9b422f92dc7d46c01c4752" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 45205 } } peers { permanent_uuid: "28daa46279c7450883d7a210d6c0b2b8" member_type: VOTER last_known_addr { host: "127.12.8.61" port: 35345 } } peers { permanent_uuid: "96f1a12b1f0b400087fed6e52f0b03d8" member_type: VOTER last_known_addr { host: "127.12.8.60" port: 44633 } } }
I20260925 15:18:37.592242 2791 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4708b049c9b422f92dc7d46c01c4752 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:18:37.593070 2623 sys_catalog.cc:564] T 00000000000000000000000000000000 P b4708b049c9b422f92dc7d46c01c4752 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:18:37.593115 2598 raft_consensus.cc:482] T 00000000000000000000000000000000 P b4708b049c9b422f92dc7d46c01c4752 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 15:18:37.593324 2598 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b4708b049c9b422f92dc7d46c01c4752 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:18:37.596143 2598 raft_consensus.cc:504] T 00000000000000000000000000000000 P b4708b049c9b422f92dc7d46c01c4752 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4708b049c9b422f92dc7d46c01c4752" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 45205 } } peers { permanent_uuid: "28daa46279c7450883d7a210d6c0b2b8" member_type: VOTER last_known_addr { host: "127.12.8.61" port: 35345 } } peers { permanent_uuid: "96f1a12b1f0b400087fed6e52f0b03d8" member_type: VOTER last_known_addr { host: "127.12.8.60" port: 44633 } }
I20260925 15:18:37.597450 2598 leader_election.cc:288] T 00000000000000000000000000000000 P b4708b049c9b422f92dc7d46c01c4752 [CANDIDATE]: Term 1 election: Requested vote from peers 28daa46279c7450883d7a210d6c0b2b8 (127.12.8.61:35345), 96f1a12b1f0b400087fed6e52f0b03d8 (127.12.8.60:44633)
I20260925 15:18:37.599663 2746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b4708b049c9b422f92dc7d46c01c4752" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "96f1a12b1f0b400087fed6e52f0b03d8"
I20260925 15:18:37.600195 2746 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 96f1a12b1f0b400087fed6e52f0b03d8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:18:37.603762 2746 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 96f1a12b1f0b400087fed6e52f0b03d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4708b049c9b422f92dc7d46c01c4752 in term 1.
I20260925 15:18:37.604439 2574 leader_election.cc:302] T 00000000000000000000000000000000 P b4708b049c9b422f92dc7d46c01c4752 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96f1a12b1f0b400087fed6e52f0b03d8, b4708b049c9b422f92dc7d46c01c4752; no voters:
I20260925 15:18:37.604964 2791 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b4708b049c9b422f92dc7d46c01c4752 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:18:37.619904 2791 raft_consensus.cc:686] T 00000000000000000000000000000000 P b4708b049c9b422f92dc7d46c01c4752 [term 1 LEADER]: Becoming Leader. State: Replica: b4708b049c9b422f92dc7d46c01c4752, State: Running, Role: LEADER
I20260925 15:18:37.621003 2791 consensus_queue.cc:237] T 00000000000000000000000000000000 P b4708b049c9b422f92dc7d46c01c4752 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4708b049c9b422f92dc7d46c01c4752" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 45205 } } peers { permanent_uuid: "28daa46279c7450883d7a210d6c0b2b8" member_type: VOTER last_known_addr { host: "127.12.8.61" port: 35345 } } peers { permanent_uuid: "96f1a12b1f0b400087fed6e52f0b03d8" member_type: VOTER last_known_addr { host: "127.12.8.60" port: 44633 } }
I20260925 15:18:37.633488 2697 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.8.60" port: 44633 } attempt: 5
W20260925 15:18:37.643334 2725 tablet.cc:2369] T 00000000000000000000000000000000 P 96f1a12b1f0b400087fed6e52f0b03d8: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 15:18:37 dist-test-slave-p468 krb5kdc[2543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349516, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.8.61@KRBTEST.COM for kudu/127.12.8.60@KRBTEST.COM
W20260925 15:18:37.647981 2577 tablet.cc:2369] T 00000000000000000000000000000000 P b4708b049c9b422f92dc7d46c01c4752: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:18:37.651413 2699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:37.634370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.8.60:44633 (local address 127.0.0.1:37086)
0925 15:18:37.634894 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0925 15:18:37.634919 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 15:18:37.635071 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:18:37.635620 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:18:37.635626 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:18:37.636158 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:18:37.636365 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:37.636377 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:37.637874 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:37.637880 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:37.639344 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:18:37.639359 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:18:37.640539 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:18:37.640545 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:18:37.640778 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:37.641446 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:18:37.641470 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:18:37.644349 (+ 2879us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:18:37.648600 (+ 4251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:37.648608 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:37.648612 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:37.649128 (+ 516us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:37.649564 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:18:37.649569 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:18:37.649573 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:18:37.649747 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:18:37.650431 (+ 684us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:18:37.650437 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:18:37.650594 (+ 157us) client_negotiation.cc:769] Sending connection context
0925 15:18:37.650742 (+ 148us) client_negotiation.cc:240] Negotiation successful
0925 15:18:37.651023 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345}
I20260925 15:18:37.652745 2774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:37.634831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37086 (local address 127.12.8.60:44633)
0925 15:18:37.635036 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:37.635042 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:37.635067 (+ 25us) server_negotiation.cc:407] Connection header received
0925 15:18:37.635205 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:37.635211 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:37.635294 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:37.635406 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:37.636556 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:37.637698 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:37.639532 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:37.640221 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:37.640347 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:37.644561 (+ 4214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:37.644567 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:37.644572 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:37.644609 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:37.648445 (+ 3836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:37.649270 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:37.649276 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:37.649280 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:37.649429 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:37.649880 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:37.649885 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:37.649889 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:37.650236 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:37.650372 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:37.652150 (+ 1778us) server_negotiation.cc:299] Negotiation successful
0925 15:18:37.652380 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 15:18:37.658696 2697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28daa46279c7450883d7a210d6c0b2b8: Bootstrap starting.
I20260925 15:18:37.665539 2697 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 28daa46279c7450883d7a210d6c0b2b8: Neither blocks nor log segments found. Creating new log.
I20260925 15:18:37.666991 2697 log.cc:824] T 00000000000000000000000000000000 P 28daa46279c7450883d7a210d6c0b2b8: Log is configured to *not* fsync() on all Append() calls
I20260925 15:18:37.676016 2697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28daa46279c7450883d7a210d6c0b2b8: No bootstrap required, opened a new log
I20260925 15:18:37.682495 2801 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4708b049c9b422f92dc7d46c01c4752 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4708b049c9b422f92dc7d46c01c4752. Latest consensus state: current_term: 1 leader_uuid: "b4708b049c9b422f92dc7d46c01c4752" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4708b049c9b422f92dc7d46c01c4752" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 45205 } } peers { permanent_uuid: "28daa46279c7450883d7a210d6c0b2b8" member_type: VOTER last_known_addr { host: "127.12.8.61" port: 35345 } } peers { permanent_uuid: "96f1a12b1f0b400087fed6e52f0b03d8" member_type: VOTER last_known_addr { host: "127.12.8.60" port: 44633 } } }
I20260925 15:18:37.682984 2801 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4708b049c9b422f92dc7d46c01c4752 [sys.catalog]: This master's current role is: LEADER
I20260925 15:18:37.688970 2697 raft_consensus.cc:348] T 00000000000000000000000000000000 P 28daa46279c7450883d7a210d6c0b2b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4708b049c9b422f92dc7d46c01c4752" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 45205 } } peers { permanent_uuid: "28daa46279c7450883d7a210d6c0b2b8" member_type: VOTER last_known_addr { host: "127.12.8.61" port: 35345 } } peers { permanent_uuid: "96f1a12b1f0b400087fed6e52f0b03d8" member_type: VOTER last_known_addr { host: "127.12.8.60" port: 44633 } }
I20260925 15:18:37.689365 2697 raft_consensus.cc:729] T 00000000000000000000000000000000 P 28daa46279c7450883d7a210d6c0b2b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28daa46279c7450883d7a210d6c0b2b8, State: Initialized, Role: FOLLOWER
I20260925 15:18:37.689910 2697 consensus_queue.cc:260] T 00000000000000000000000000000000 P 28daa46279c7450883d7a210d6c0b2b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4708b049c9b422f92dc7d46c01c4752" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 45205 } } peers { permanent_uuid: "28daa46279c7450883d7a210d6c0b2b8" member_type: VOTER last_known_addr { host: "127.12.8.61" port: 35345 } } peers { permanent_uuid: "96f1a12b1f0b400087fed6e52f0b03d8" member_type: VOTER last_known_addr { host: "127.12.8.60" port: 44633 } }
I20260925 15:18:37.711601 2701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:37.621342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51384 (local address 127.12.8.61:35345)
0925 15:18:37.621507 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:37.621514 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:37.673306 (+ 51792us) server_negotiation.cc:407] Connection header received
0925 15:18:37.673386 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:37.673392 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:37.673486 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:37.673595 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:37.675914 (+ 2319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:37.677043 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:37.678956 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:37.679710 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:37.679992 (+ 282us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:37.697051 (+ 17059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:37.697057 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:37.697070 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:37.697119 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:37.701456 (+ 4337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:37.705333 (+ 3877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:37.705339 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:37.705342 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:37.705503 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:37.706268 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:37.706274 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:37.706277 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:37.706696 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:37.706973 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:37.710759 (+ 3786us) server_negotiation.cc:299] Negotiation successful
0925 15:18:37.711086 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 15:18:37.714720 2774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:37.671219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37100 (local address 127.12.8.60:44633)
0925 15:18:37.671415 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:37.671422 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:37.694980 (+ 23558us) server_negotiation.cc:407] Connection header received
0925 15:18:37.695237 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:37.695244 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:37.695363 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:37.695438 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:37.697244 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:37.698406 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:37.700519 (+ 2113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:37.701244 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:37.701348 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:37.705335 (+ 3987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:37.705342 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:37.705350 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:37.705393 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:37.708910 (+ 3517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:37.710406 (+ 1496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:37.710411 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:37.710415 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:37.710559 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:37.711361 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:37.711366 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:37.711369 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:37.711696 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:37.711940 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:37.714093 (+ 2153us) server_negotiation.cc:299] Negotiation successful
0925 15:18:37.714349 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 15:18:37.722800 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:37.620650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57790 (local address 127.12.8.62:45205)
0925 15:18:37.620851 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:37.620857 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:37.671841 (+ 50984us) server_negotiation.cc:407] Connection header received
0925 15:18:37.672883 (+ 1042us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:37.672892 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:37.672994 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:37.673093 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:37.692719 (+ 19626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:37.693834 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:37.703417 (+ 9583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:37.704100 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:37.704226 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:37.713758 (+ 9532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:18:37.713768 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:18:37.713772 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:18:37.713824 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:18:37.718621 (+ 4797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:37.719705 (+ 1084us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:37.719712 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:37.719718 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:37.719962 (+ 244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:18:37.720513 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:18:37.720521 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:18:37.720527 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:18:37.720968 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:18:37.721292 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:37.721956 (+ 664us) server_negotiation.cc:299] Negotiation successful
0925 15:18:37.722295 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 15:18:37.726153 2814 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28daa46279c7450883d7a210d6c0b2b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4708b049c9b422f92dc7d46c01c4752" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 45205 } } peers { permanent_uuid: "28daa46279c7450883d7a210d6c0b2b8" member_type: VOTER last_known_addr { host: "127.12.8.61" port: 35345 } } peers { permanent_uuid: "96f1a12b1f0b400087fed6e52f0b03d8" member_type: VOTER last_known_addr { host: "127.12.8.60" port: 44633 } } }
I20260925 15:18:37.728034 2814 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28daa46279c7450883d7a210d6c0b2b8 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:18:37.729560 2697 sys_catalog.cc:564] T 00000000000000000000000000000000 P 28daa46279c7450883d7a210d6c0b2b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:18:37.729594 2672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b4708b049c9b422f92dc7d46c01c4752" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "28daa46279c7450883d7a210d6c0b2b8"
I20260925 15:18:37.730087 2672 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 28daa46279c7450883d7a210d6c0b2b8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:18:37.707430 2556 master_runner.cc:186] Error getting master registration for 127.12.8.61:35345: OK, instance_id { permanent_uuid: "28daa46279c7450883d7a210d6c0b2b8" instance_seqno: 1790349516906934 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 15:18:37.731271 2556 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260925 15:18:37.731725 2812 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b4708b049c9b422f92dc7d46c01c4752: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:18:37.731910 2812 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:18:37.732070 2813 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:18:37.733599 2672 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 28daa46279c7450883d7a210d6c0b2b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4708b049c9b422f92dc7d46c01c4752 in term 1.
I20260925 15:18:37.734073 2813 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:18:37.741981 2746 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 96f1a12b1f0b400087fed6e52f0b03d8 [term 1 FOLLOWER]: Refusing update from remote peer b4708b049c9b422f92dc7d46c01c4752: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:18:37.742249 2672 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 28daa46279c7450883d7a210d6c0b2b8 [term 1 FOLLOWER]: Refusing update from remote peer b4708b049c9b422f92dc7d46c01c4752: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:18:37.743183 2791 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b4708b049c9b422f92dc7d46c01c4752 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28daa46279c7450883d7a210d6c0b2b8" member_type: VOTER last_known_addr { host: "127.12.8.61" port: 35345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:18:37.743991 2791 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b4708b049c9b422f92dc7d46c01c4752 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96f1a12b1f0b400087fed6e52f0b03d8" member_type: VOTER last_known_addr { host: "127.12.8.60" port: 44633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:18:37.765291 2777 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96f1a12b1f0b400087fed6e52f0b03d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4708b049c9b422f92dc7d46c01c4752. Latest consensus state: current_term: 1 leader_uuid: "b4708b049c9b422f92dc7d46c01c4752" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4708b049c9b422f92dc7d46c01c4752" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 45205 } } peers { permanent_uuid: "28daa46279c7450883d7a210d6c0b2b8" member_type: VOTER last_known_addr { host: "127.12.8.61" port: 35345 } } peers { permanent_uuid: "96f1a12b1f0b400087fed6e52f0b03d8" member_type: VOTER last_known_addr { host: "127.12.8.60" port: 44633 } } }
I20260925 15:18:37.765626 2777 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96f1a12b1f0b400087fed6e52f0b03d8 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:18:37.769774 2814 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28daa46279c7450883d7a210d6c0b2b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4708b049c9b422f92dc7d46c01c4752. Latest consensus state: current_term: 1 leader_uuid: "b4708b049c9b422f92dc7d46c01c4752" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4708b049c9b422f92dc7d46c01c4752" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 45205 } } peers { permanent_uuid: "28daa46279c7450883d7a210d6c0b2b8" member_type: VOTER last_known_addr { host: "127.12.8.61" port: 35345 } } peers { permanent_uuid: "96f1a12b1f0b400087fed6e52f0b03d8" member_type: VOTER last_known_addr { host: "127.12.8.60" port: 44633 } } }
I20260925 15:18:37.770112 2814 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28daa46279c7450883d7a210d6c0b2b8 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:18:37.778087 2801 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4708b049c9b422f92dc7d46c01c4752 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b4708b049c9b422f92dc7d46c01c4752" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4708b049c9b422f92dc7d46c01c4752" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 45205 } } peers { permanent_uuid: "28daa46279c7450883d7a210d6c0b2b8" member_type: VOTER last_known_addr { host: "127.12.8.61" port: 35345 } } peers { permanent_uuid: "96f1a12b1f0b400087fed6e52f0b03d8" member_type: VOTER last_known_addr { host: "127.12.8.60" port: 44633 } } }
I20260925 15:18:37.778404 2801 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4708b049c9b422f92dc7d46c01c4752 [sys.catalog]: This master's current role is: LEADER
I20260925 15:18:37.778676 2801 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4708b049c9b422f92dc7d46c01c4752 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b4708b049c9b422f92dc7d46c01c4752" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4708b049c9b422f92dc7d46c01c4752" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 45205 } } peers { permanent_uuid: "28daa46279c7450883d7a210d6c0b2b8" member_type: VOTER last_known_addr { host: "127.12.8.61" port: 35345 } } peers { permanent_uuid: "96f1a12b1f0b400087fed6e52f0b03d8" member_type: VOTER last_known_addr { host: "127.12.8.60" port: 44633 } } }
I20260925 15:18:37.778913 2801 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4708b049c9b422f92dc7d46c01c4752 [sys.catalog]: This master's current role is: LEADER
I20260925 15:18:37.780555 2814 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28daa46279c7450883d7a210d6c0b2b8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b4708b049c9b422f92dc7d46c01c4752" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4708b049c9b422f92dc7d46c01c4752" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 45205 } } peers { permanent_uuid: "28daa46279c7450883d7a210d6c0b2b8" member_type: VOTER last_known_addr { host: "127.12.8.61" port: 35345 } } peers { permanent_uuid: "96f1a12b1f0b400087fed6e52f0b03d8" member_type: VOTER last_known_addr { host: "127.12.8.60" port: 44633 } } }
I20260925 15:18:37.780814 2814 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28daa46279c7450883d7a210d6c0b2b8 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:18:37.782353 2820 mvcc.cc:205] Tried to move back new op lower bound from 7333271624660987904 to 7333271624181280768. Current Snapshot: MvccSnapshot[applied={T|T < 7333271624660987904}]
I20260925 15:18:37.782949 2821 mvcc.cc:205] Tried to move back new op lower bound from 7333271624660987904 to 7333271624181280768. Current Snapshot: MvccSnapshot[applied={T|T < 7333271624660987904}]
I20260925 15:18:37.786454 2813 catalog_manager.cc:1347] Generated new cluster ID: ad22fea3daa04007a4b31dce1310676d
I20260925 15:18:37.786550 2813 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:18:37.787468 2777 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96f1a12b1f0b400087fed6e52f0b03d8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b4708b049c9b422f92dc7d46c01c4752" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4708b049c9b422f92dc7d46c01c4752" member_type: VOTER last_known_addr { host: "127.12.8.62" port: 45205 } } peers { permanent_uuid: "28daa46279c7450883d7a210d6c0b2b8" member_type: VOTER last_known_addr { host: "127.12.8.61" port: 35345 } } peers { permanent_uuid: "96f1a12b1f0b400087fed6e52f0b03d8" member_type: VOTER last_known_addr { host: "127.12.8.60" port: 44633 } } }
I20260925 15:18:37.787693 2777 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96f1a12b1f0b400087fed6e52f0b03d8 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:18:37.828401 2813 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:18:37.830592 2813 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:18:37.840365 2834 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 15:18:37.849753 2813 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b4708b049c9b422f92dc7d46c01c4752: Generated new TSK 0
I20260925 15:18:37.850863 2813 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 15:18:37.952862 2701 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:18:37.942994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51394 (local address 127.12.8.61:35345)
0925 15:18:37.943194 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:37.943201 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:37.950798 (+ 7597us) server_negotiation.cc:407] Connection header received
0925 15:18:37.951247 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:37.951253 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:37.951335 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:37.951408 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:37.952626 (+ 1218us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51394: BlockingRecv error: recv got EOF from 127.0.0.1:51394 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260925 15:18:37.953203 2626 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:18:37.943576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57798 (local address 127.12.8.62:45205)
0925 15:18:37.943778 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:37.943784 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:37.950838 (+ 7054us) server_negotiation.cc:407] Connection header received
0925 15:18:37.951156 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:37.951162 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:37.951254 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:37.951349 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:37.952935 (+ 1586us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57798: BlockingRecv error: recv got EOF from 127.0.0.1:57798 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260925 15:18:37.955005 2774 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:18:37.951057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37104 (local address 127.12.8.60:44633)
0925 15:18:37.951687 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:37.951696 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:37.953484 (+ 1788us) server_negotiation.cc:407] Connection header received
0925 15:18:37.953715 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:37.953721 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:37.953828 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:37.953898 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:37.954775 (+ 877us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37104: BlockingRecv error: recv got EOF from 127.0.0.1:37104 (error 108)
Metrics: {"server-negotiator.queue_time_us":452}
W20260925 15:18:37.964154 2701 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:18:37.961965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51410 (local address 127.12.8.61:35345)
0925 15:18:37.962146 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:37.962154 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:37.962650 (+ 496us) server_negotiation.cc:407] Connection header received
0925 15:18:37.962886 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:37.962892 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:37.963024 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:37.963134 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:37.963898 (+ 764us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51410: BlockingRecv error: recv got EOF from 127.0.0.1:51410 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260925 15:18:37.970638 2701 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:18:37.968684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51426 (local address 127.12.8.61:35345)
0925 15:18:37.968844 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:37.968850 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:37.969285 (+ 435us) server_negotiation.cc:407] Connection header received
0925 15:18:37.969458 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:37.969463 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:37.969552 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:37.969666 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:37.970434 (+ 768us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51426: BlockingRecv error: recv got EOF from 127.0.0.1:51426 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260925 15:18:37.977907 2701 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:18:37.975801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51440 (local address 127.12.8.61:35345)
0925 15:18:37.975944 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:37.975949 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:37.976397 (+ 448us) server_negotiation.cc:407] Connection header received
0925 15:18:37.976573 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:37.976578 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:37.976704 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:37.976813 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:37.977694 (+ 881us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51440: BlockingRecv error: recv got EOF from 127.0.0.1:51440 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260925 15:18:37.990301 2701 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:18:37.987984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51452 (local address 127.12.8.61:35345)
0925 15:18:37.988188 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:37.988195 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:37.988680 (+ 485us) server_negotiation.cc:407] Connection header received
0925 15:18:37.988970 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:37.988976 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:37.989081 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:37.989198 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:37.990020 (+ 822us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51452: BlockingRecv error: recv got EOF from 127.0.0.1:51452 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
W20260925 15:18:38.004339 2701 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:18:38.001851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51460 (local address 127.12.8.61:35345)
0925 15:18:38.002076 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:38.002083 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:38.002855 (+ 772us) server_negotiation.cc:407] Connection header received
0925 15:18:38.003109 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:38.003115 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:38.003216 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:38.003327 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:38.004106 (+ 779us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51460: BlockingRecv error: recv got EOF from 127.0.0.1:51460 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
W20260925 15:18:38.026268 2701 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:18:38.023917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51466 (local address 127.12.8.61:35345)
0925 15:18:38.024123 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:38.024130 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:38.024646 (+ 516us) server_negotiation.cc:407] Connection header received
0925 15:18:38.024864 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:38.024870 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:38.024964 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:38.025081 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:38.025984 (+ 903us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51466: BlockingRecv error: recv got EOF from 127.0.0.1:51466 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
W20260925 15:18:38.063977 2701 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:18:38.061954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51470 (local address 127.12.8.61:35345)
0925 15:18:38.062164 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:38.062170 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:38.062615 (+ 445us) server_negotiation.cc:407] Connection header received
0925 15:18:38.062790 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:38.062796 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:38.062886 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:38.063002 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:38.063724 (+ 722us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51470: BlockingRecv error: recv got EOF from 127.0.0.1:51470 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260925 15:18:38.133857 2701 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:18:38.131854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51472 (local address 127.12.8.61:35345)
0925 15:18:38.132089 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:38.132095 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:38.132390 (+ 295us) server_negotiation.cc:407] Connection header received
0925 15:18:38.132612 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:38.132616 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:38.132689 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:38.132781 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:38.133643 (+ 862us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51472: BlockingRecv error: recv got EOF from 127.0.0.1:51472 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
W20260925 15:18:38.267462 2701 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:18:38.265474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51478 (local address 127.12.8.61:35345)
0925 15:18:38.265658 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:38.265665 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:38.266003 (+ 338us) server_negotiation.cc:407] Connection header received
0925 15:18:38.266172 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:38.266177 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:38.266270 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:38.266371 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:38.267171 (+ 800us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51478: BlockingRecv error: recv got EOF from 127.0.0.1:51478 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 15:18:38.511936 2788 catalog_manager.cc:1259] Loaded cluster ID: ad22fea3daa04007a4b31dce1310676d
I20260925 15:18:38.512037 2788 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 96f1a12b1f0b400087fed6e52f0b03d8: loading cluster ID for follower catalog manager: success
I20260925 15:18:38.514397 2788 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 96f1a12b1f0b400087fed6e52f0b03d8: acquiring CA information for follower catalog manager: success
I20260925 15:18:38.516077 2788 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 96f1a12b1f0b400087fed6e52f0b03d8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260925 15:18:38.528920 2701 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:18:38.527053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51486 (local address 127.12.8.61:35345)
0925 15:18:38.527307 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:38.527312 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:38.527408 (+ 96us) server_negotiation.cc:407] Connection header received
0925 15:18:38.527636 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:38.527641 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:38.527730 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:38.527828 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:18:38.528702 (+ 874us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51486: BlockingRecv error: recv got EOF from 127.0.0.1:51486 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
I20260925 15:18:38.844807 2834 catalog_manager.cc:1259] Loaded cluster ID: ad22fea3daa04007a4b31dce1310676d
I20260925 15:18:38.844964 2834 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 28daa46279c7450883d7a210d6c0b2b8: loading cluster ID for follower catalog manager: success
I20260925 15:18:38.847725 2834 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 28daa46279c7450883d7a210d6c0b2b8: acquiring CA information for follower catalog manager: success
I20260925 15:18:38.849489 2834 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 28daa46279c7450883d7a210d6c0b2b8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 15:18:39.053730 2900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:39.045943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51492 (local address 127.12.8.61:35345)
0925 15:18:39.046685 (+ 742us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:39.046692 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:39.047004 (+ 312us) server_negotiation.cc:407] Connection header received
0925 15:18:39.047186 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:39.047193 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:39.047399 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:39.047557 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:18:39.048046 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:39.049255 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:39.050760 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:39.051547 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:39.051680 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:39.052218 (+ 538us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:18:39.052992 (+ 774us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:18:39.053140 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:39.053253 (+ 113us) server_negotiation.cc:299] Negotiation successful
0925 15:18:39.053355 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":420,"threads_started":1}
I20260925 15:18:39.066130 2906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:39.059718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37118 (local address 127.12.8.60:44633)
0925 15:18:39.060343 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:39.060347 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:39.060380 (+ 33us) server_negotiation.cc:407] Connection header received
0925 15:18:39.060482 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:39.060485 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:39.060622 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:39.060755 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:18:39.061224 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:39.061910 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:39.063845 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:39.064262 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:39.064370 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:39.064960 (+ 590us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:18:39.065467 (+ 507us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:18:39.065573 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:39.065723 (+ 150us) server_negotiation.cc:299] Negotiation successful
0925 15:18:39.065790 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":393,"threads_started":1}
I20260925 15:18:39.087174 2912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:39.072412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57812 (local address 127.12.8.62:45205)
0925 15:18:39.079162 (+ 6750us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:39.079168 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:39.079188 (+ 20us) server_negotiation.cc:407] Connection header received
0925 15:18:39.079301 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:39.079320 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:39.079555 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:39.080348 (+ 793us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:18:39.080580 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:39.081822 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:39.084123 (+ 2301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:39.084739 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:39.085379 (+ 640us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:39.085562 (+ 183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:18:39.085993 (+ 431us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:18:39.086694 (+ 701us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:39.086751 (+ 57us) server_negotiation.cc:299] Negotiation successful
0925 15:18:39.086836 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6617,"thread_start_us":6514,"threads_started":1}
I20260925 15:18:39.087284 2906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:39.079048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37134 (local address 127.12.8.60:44633)
0925 15:18:39.079228 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:39.079234 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:39.079957 (+ 723us) server_negotiation.cc:407] Connection header received
0925 15:18:39.080281 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:39.080288 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:39.080391 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:39.080451 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:18:39.081410 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:39.082509 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:39.084999 (+ 2490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:39.085624 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:39.085722 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:39.086536 (+ 814us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:18:39.086650 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:18:39.086709 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:39.086932 (+ 223us) server_negotiation.cc:299] Negotiation successful
0925 15:18:39.087011 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 15:18:39.087337 2900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:18:39.072412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51502 (local address 127.12.8.61:35345)
0925 15:18:39.072635 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0925 15:18:39.072644 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:18:39.078786 (+ 6142us) server_negotiation.cc:407] Connection header received
0925 15:18:39.080022 (+ 1236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:18:39.080029 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:18:39.080118 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:18:39.080193 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:18:39.081166 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:39.082387 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:39.085003 (+ 2616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:18:39.085640 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:18:39.085722 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:18:39.086535 (+ 813us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:18:39.086650 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:18:39.086709 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 15:18:39.086950 (+ 241us) server_negotiation.cc:299] Negotiation successful
0925 15:18:39.087022 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 15:18:39.093672 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 2556
I20260925 15:18:39.094826 2618 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:18:39.205754 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 2556
I20260925 15:18:39.230727 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 2630
I20260925 15:18:39.232038 2692 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:18:39.343351 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 2630
I20260925 15:18:39.364341 12320 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBtce3V/build/asan/bin/kudu pid 2704
I20260925 15:18:39.365495 2766 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:18:39.471590 12320 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBtce3V/build/asan/bin/kudu with pid 2704
2026-09-25T15:18:39Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4884 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9868 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (357066 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST